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A COMPARISON OF THE USE OF DIRECT AND INDIRECT 
PHRASING IN PERSONALITY QUESTIONNAIRES 


CHAPTER I 


INTRODUCTION 


HEN Woodworth (28) constructed 
Witte Personal Data Sheet during the 
first World War, he started a fashion in 
personality questionnaires which has 
had an enormous influence on psycho- 
logical and educational research and 
practice. In the course of the last thirty 
years, the popularity of paper and pencil 
personality tests has been impressively 
proved not only by their widespread use 
but also by the great number of studies 
devoted to their construction, stand- 
ardization, reliability and validity. 

One consequence of the extensive re- 
searches on conventional . personality 
questionnaires has been the develop- 
ment, today, of instruments which are 
usually conceded to be highly reliable. 
In terms of validity, however, results 
have been far less decisive and promis- 
ing. Material on the outcome of studies 
of the validity of personality inventories 
has been reviewed recently by the present 
author (4, 5), Who summarized the valida- 
tion data published between 1919 and 
1946 on tests of the Woodworth, Bern- 
reuter, and Thurstone type. The find- 
ings were as follows: 

1. When used to distinguish between 
groups of problem and nonproblem chil- 
dren, personality questionnaires  dis- 
criminated significantly in two experi- 
ments, discriminated with a questionable 
degree of significance in one, and were 
non-discriminative in six. 

2. When utilized with delinquents, 
the questionnaires differentiated  sig- 
nificantly between delinquent and non- 
delinquent groups in fifteen, to a ques- 
tionable degree in six, and were non- 


differentiating in thirteen investigations. 

3. When employed for purposes of 
psychiatric and psychological diagnosis, 
the inventories were adequate for group 
diagnosis in thirty-six, questionably ade- 
quate in nine, and inadequate in twenty- 
two studies. 

4. When scores obtained by individ- 
uals taking personality questionnaires 
were compared to ratings given these 
individuals by their friends, associates, 
teachers, or supervisors, they were found 
to be closely related in twelve, related 
to a questionable degree in ten, and to a 
low degree in twenty-two investigations. 

5. When _intercorrelations between 
subjects’ scores on two similar personal- 
ity schedules purporting to measure the 
same traits were studied, they were found 
to be between .70 and 1.00 in nine, be- 
tween .40 and .69 in eighteen, and be- 
tween .oo and .39 in twenty-eight in- 
vestigations. 

6. When a review was made of experi- 
ments specifically designed to discover 
whether subjects taking personality 
questionnaires over-rated themselves or 
lied about their responses, it was discov- 
ered that thirty-six studies indicated that 
over-rating did take place, while only six 
showed that it did not. 

On the basis of these findings, it was 
concluded that “group-administered pa- 
per and pencil personality questionnaires 
are of dubious value in distinguishing 
between groups of adjusted and malad- 
justed individuals, and . . . are of much 
less value in the diagnosis of individual 
adjustment or personality traits” (4, p. 


426). 
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Personality inventories, however, have 
definite practical advantages. They are 
easy to administer and score; give results 
that may be objectively and statistically 
treated; may serve as a formalized inter- 
view; and at least give a picture of what 
the subjects say they feel and do. Because 
of these advantages, personality tests 
have maintained their popularity despite 
the recent growth of projective and non- 
paper-and-pencil techniques of assessing 
personality. 

Is it possible to re-fashion personality 
questionnaires to eliminate some of their 
worst drawbacks and yet retain the ad- 
vantages of ease of administration, scor- 
ing and interpretation? Several attempts 
in this direction have been made since 
Woodworth’s original Personal Data 
Sheet was published. Thus, Laird (12) 
modified the YES-NO method of re- 
sponse by providing for several possible 
answers to each question. Bernreuter 
(3) introduced the inventory method of 
scoring a subject’s answers to a single set 
of questions so that he receives ratings 
on several different traits. Maller (14) 
suggested stating the questions in a less 
personal form. Humm and Wadsworth 
(9), Washburne (27), and McKinley and 
Hathaway (7, 15) developed methods of 
estimating dishonesty and correcting for 
it. McKinley and Hathaway (7) employed 
a card-sorting, individual administrating, 
instead of the usual pencil-marking 
group administration. Adams and Lepley 
(1) favored innocuous questions that pre- 
sumably would not embarrass respond- 
ents. Shipley and his associates (19) used 
a forced choice technique in the Per- 
sonal Inventory. 

All the foregoing modifications of the 
original Personal Data Sheet have added 
to the usefulness of personality question- 
naires, but none has yet resulted in an 
instrument whose validity is unques- 
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tioned. Further experimentation in this 
connection would consequently seem de. 
sirable at present. 


SPECIFIC PURPOSES OF THE PRESENT 
EXPERIMENT 


The purpose of the present experiment 
was to study the effect of indirect phras- 
ing in personality questionnaires. The 
question to be investigated was: Since 
the doubtful validity of conventional 
personality schedules may be partly at- 
tributable to respondents often discov- 
ering the intent of the questions, deter- 
mining what the “good” or “correct” 
answers are, and then giving “good” re- 
sponses, is it possible for inventory ques. 
tions to be indirectly and less transpar- 
ently stated so that schedules using them 
will be more effective for diagnostic and 
predictive purposes than are the present 
direct test forms? 

Attempts to employ indirect ques- 
tions in personality schedules are not en- 
tirely new. Sweet (23), Spencer (22), and 
Tuckman (26) devised questionnaires 
which included some indirectly phrased 
items. None of these investigators, how- 
ever, specifically tried to compare the 
efficacy of direct and indirect personality 
questions utilized on the same subjects. 
The point of departure of the present 
experiment was that it studied the use 
of equated indirect and direct person- 
ality items in the same test battery; in- 
vestigated the effect of these two kinds 
of personality questions on a_ behavior 
problem and a non-preblem group of 
children; and tried to determine what 
was the effect of giving the direct and 
indirect questions in three different or- 
ders of presentation—a direct-indirect, 
alternate, and _ indirect-direct order. 
There were thus three major variables 
operating in the experiment: two 
groups; two kinds of tests; and three dif- 
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ferent orders of presentation. 

The specific hypothesis of the experi- 
ment was: If conventional direct per- 
sonality test forms, as well as indirect 
schedules derived from the same ques- 
tions included on the direct forms, are 
given in a single battery to behavior 
problem and non-problem children: (1) 
the indirect test scores will be signifi- 
cantly different from the direct test 
scores; (2) groups of problem and non- 
problem children will react differently 
to the indirect and direct tests; and (3) 
presentation of the questions in three 


different orders—an indirect-direct, an 
alternate, and a direct-indirect order— 
will have a significant effect on the de- 
gree of test sophistication shown by the 
children in their responses. 

Another way of stating the major 
premise of the experiment is to say that, 
under the given testing conditions, mak- 
ing use of the three major variables of 
tests, groups, and orders, it is hypothe- 
sized that significant differences in the 
subjects’ mean test scores will result from 
the influence of every one of these vari- 
ables. 
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CHAPTER II 


EXPERIMENTAL MATERIALS, SUBJECTS AND DESIGN 


MATERIALS 


I* ORDER to obtain a schedule of direct 
questions that would be representa- 
tive of the conventional tests widely 
used with children today, a frequency 
count was made of the items currently 
employed in the Woodworth-Mathews 
(28), Loofbourow-Keys (13), California 
(25), Pintner Aspects of Personality (17), 
Rogers (18), Hildreth (8), Maller (14), 
and Symonds (24) personality inven- 
tories. The thirty-six most popular ques- 
tions were culled from these tests and 
were used as the basic direct question- 
naire (Test D-1) employed in the experi- 
ment. (The complete questionnaire is 
printed in the Appendix.) 

In order to see whether the manner of 
phrasing the direct questions them- 
selves was of significant import, a second 
direct form (Test D-2) was constructed. 
In Test D-g the same thirty-six ques- 
tions used in Test D-1 were rephrased so 
that while D-1 items asked about a nega- 
tive symptom, D-2 items asked about a 
positive one. Thus, where the third ques- 
tion on D-1 read, “I cry VERY 
OFTEN PRETTY OFTEN SELDOM 
NEVER,” the equivalent item on D-2 
read, “I am free from crying VERY 
OFTEN PRETTY OFTEN SELDOM 
NEVER.” Whereas Test D-1 was scored 
so that the subjects received one point 
for each item answered VERY OFTEN 
or PRETTY OFTEN and no points for 
each one answered SELDOM or NEVER, 
Test D-2 was scored so that subjects re- 
ceived one point for each SELDOM or 
NEVER response and no points for each 
VERY OFTEN or PRETTY OFTEN 
answer. 

The first of the two indirect test forms 


used in the experiment (Test 1-1) wa 
constructed by taking the same thiry. 
six questions utilized in Test D-1 ani 
rephrasing them so that the subjects wer 
asked to report how queer they consid. 
ered other children who had symptom 
like those described in the D-1 iteny 
Thus, the I-1 version of the question o 
crying was, “Children who often cry are 
VERY QUEER PRETTY QUEER 4 
LITTLE QUEER NOT AT ALL 
QUEER.” The theory behind this form 
of indirect questioning was that chil 
dren possessing a symptom like frequent 
crying would tend to see other children 
who had this symptom as being A LIT. 
TLE QUEER or NOT: AT ALL 
QUEER rather than as VERY QUEER 
or PRETTY QUEER. Preliminary ex. 
perimentation showed, however, that the 
I-1 items were more meaningfully evalu: 
ated when a score of one point was given 
to each NOT AT ALL QUEER respons 
and a score of no points to the VERY 
QUEER, PRETTY QUEER, and A 
LITTLE QUEER responses. This, con- 
sequently, was the scoring system em- 
ployed for Test I-1. 

The second indirect questionnaire em: 
ployed in the experiment (Test I-2) was 
constructed by using, once again, the 
original thirty-six questions of Test D-\, 
but this time asking the subjects how 
many other children they thought pos 
sessed the symptoms described in the D- 
items. Thus, where on D-1 the subject 
was asked how often he cried, on the 
equivalent I-2 item he was asked to re 
spond to the question, “VERY MANY 
PRETTY MANY FEW NO children 
often cry.” The theory behind the I 
rephrasings of the D-1 items was that the 
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subjects who actually possessed symptoms 
like crying frequently would tend to re- 
ort that there were VERY MANY or 
PRETTY MANY children who also had 
these symptoms; while subjects who ac- 
tually exhibited the symptoms seldom 
or never would tend to report FEW or 
NO other children as having them. Con- 
sequently, on I-g items a score of one 
point was given for the VERY MANY 
or PRETTY MANY responses and a 
score of no points for the FEW or NO 
answers. 

All four test forms, D-1, D-2, I-1 and 
I-2, were bound together in a single 
booklet and administered to the sub- 
jects at one sitting, in the manner ex- 
plained below. The only additional ma- 
terial employed in the experiment was 
the Pintner Intermediate Test (16), a 
standard group test of intelligence 
which was routinely administered to the 
subjects several weeks before the ad- 
ministration of the personality tests. 


SUBJECTS 


The subjects of the experiment con- 
sisted of two groups of boys from regular 
seventh and eighth grade classes of three 
New York City junior high schools: 

1. The behavior problem children—of 
which there were forty—were known as 
serious problem cases in their schools. 
This meant that each pupil had been 
brought to the attention of the school 
principal on several occasions and had 
been referred to the school psychologist 
for special guidance. While known as 
problem children, these forty subjects 
were not among the worst problems in 
the New York City schools, and were 
permitted to participate in regular school 
classes rather than special or segregated 
ones. There is little reason to assume 
that the problem group was a homo- 
geneous one, since the offences against 


school discipline for which the members 
of the group were brought to the atten- 
tion of the authorities were quite varied. 
Moreover, even when similar offences 
were committed, the reasons for them 
may have ranged from serious neurosis 
to a rather healthy aggressiveness. 

2. The non-problem group consisted 
of 221 unselected school children in regu- 
lar classes who were having no special 
behavior difficulties in school. 

The test booklets were administered 
to the children by their own teachers 
during their home-room periods. All the 
children in each class were tested, the 
problem along with the non-problem 
subjects. Segregation of the two groups 
of papers was made only after they had 
been handed in and scored. 


DESIGN OF THE EXPERIMENT 


In designing the experiment, several 
considerations had to be taken into ac- 
count: 

1. The four personality test forms had 
to be given at equally spaced time in- 
tervals, and at intervals which would 
minimize the factor of intervening oc- 
currences. This was accomplished by 
binding them together in a single test 
booklet which could be answered by all 
the subjects at one sitting. Since there 
were only 144 brief questions all told, 
the element of test fatigue was negligible. 

2. The tests had to be presented in 
three different orders: (a) all the indirect 
questions followed by all the direct ones; 
(b) a page of indirect alternated with a 
page of direct questions; and (c) all the 
direct questions followed by all the in- 
direct ones. This was done by dividing 
each test into four pages, each page con- 
taining nine printed questions. The 
pages were then assembled and bound in 
the three required. orders, so that one- 
third of the children could be given in- 














direct-direct, one-third alternate, and 
one-third direct-indirect booklets. 

3. Perseveration tendencies on the 
part of the subjects had to be minimized. 
This was accomplished by assembling 
the sixteen pages of each booklet so that 
two pages of the same test form never 
followed each other. Thus, in the in- 
direct-direct booklets, there would be a 
page of I-2 questions alternated with a 
page of I-1 questions for the first half 
of the battery; and a page of D-1 alter- 
nated with a page of D-2 questions for 
the last half of the booklet. 

4- Randomization of the individual 
questions in each test had to be effected. 
This was done by numbering them on 
separate slips of paper, mixing them 
thoroughly in a box, and then picking 
them out individually and printing 
them in the order thus selected. 

5- Randomization of the individuals 
and equalization of the numbers of sub- 
jects taking the three test orders had to 
be arranged. This was accomplished by 
stacking the booklets so that an indirect- 
direct battery was followed by an aler- 
nate one, which was in turn followed by 
a direct-indirect one, and so on. The 
booklets were then distributed to the 
subjects so that chance factors deter- 
mined which test order any individual 
child received; while, in every class 
tested, approximately one-third of the 
children received a direct-indirect, one- 
third an alternative, and one-third an 
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indirect-direct booklet. The fact tha 
children who sat near each other had 
diferently arranged booklets also meant 
that attempts at copying were frustrated, 

6. Environmental factors in the school 
situation had to be similar for both 
problem and non-problem subjects in 
order to effect maximal experimental 
control. This was done (as outlined 
above) by selecting problem children 
who attended regular school classes and 
who were therefore not responding un. 
der segregated or other conditions which 
might have effected the testing environ. 
ment. 

7. Randomization of individuals and 
equalization of numbers of subjects tak. 
ing the three test orders had to be a. 
complished for problem children as well 
as non-problem, children involved in the 
experiment. This was arranged by hav- 
ing the problem subjects answer the test 
battery in their regular classroom groups, 
exactly as did the non-problem children; 
so that no distinction was made between 
the two groups in the course of the test 
administration. 

In brief: the experiment was so de- 
signed as to permit the four personality 
test forms to be given to the two ex 
perimental groups in three different 
orders of presentation without any sys- 
tematic biases as to the arrangement ol 
the questions, the subjects, the tests, the 
groups, or the orders. 
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CHAPTER III 


THE DATA AND THEIR TREATMENT 


HE BASIC data on the mean scores and 

"T variabitity obtained by 221 non- 
problem and 40 problem children on 
the four personality test forms used in 
the experiment are presented in Table 1. 
From Table 1 it would appear that, 
for both the problem and non-problem 
groups: (1) Tests I-1 and I-2 differed 


(20, 21), seemed to be the most practical 
statistical method of treating the data 
of Table 1. Application of Bartlett’s (2) 
test for the homogeneity of variance to 
the data revealed a sufficient degree of 
homogeneity to make the use of analysis 
of variance feasible, so it was accordingly 
employed. 


TABLE 1 


Mean scores and standard deviations. 221 non-problem and 40 problem children taking each 
four forms of a frsonality test in three orders of presentation 








Mean Score and Standard Deviation 





Order of 


Presentation Test I-1 


Test I-2 


Test D-1 Test D-2 





M S.D. N M 


S.D. N M_ 5S.D. N M_ SD. 





No -Problem 
Children 
Indirect- 
direct 
Alternate 
Direct- 
indirect 


Problem 
Children 
Indirect- 
direct 13 
Alternate 14 
Direct- 
indirect 13 


6.07 13 
9.21 14 


16.53 
17.21 


12.46 
12.21 
16.85 


10.32 13 ~8.69 


6.84 13 
4-47 14 


6.92 
6.36 


5.11 13 6.54 
4-71 14 6.64 


4.96 
5.12 


7.05 13 3.62 3.65 13. 4.07 3.04 





from each other; (2) Tests D-1 and D-2 
did not differ; (g) the two indirect tests 
differed from the direct ones; and (4) 
some order differences existed. The ques- 
tion is: Are these differences statistically 
significant? 

Since three main variables were in- 
volved in the experiment—groups, tests, 
and orders—and since considerable inter- 
action among these variables was pos- 
sible, the analysis of variance technique, 
devised by Fisher (6) and ably ex- 
pounded in this country by Snedecor 


* All calculations were carried out to the third decimal place; but for ease of reading, only two 
decimal places are used in this and all succeeding tables. 


THE PROBLEMS TREATED 

Problem 1a. Were the subjects’ mean 
scores on the four personality tests sig- 
nificantly affected by: Form of the test? 
Composition of the group taking the 
test? Order of presentation of the ques- 
tions? 

Table 2 presents the basic analysis of 
variance applied to the data of Table 1. 
Table 2 may be interpreted as follows: 
If the Mean Square of the individual 
scores within subclasses (37.09) is taken as 
the best estimate of the error variance 
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TABLE 2 


Analysis of variance of mean scores. 221 non-problem and 40 problem children taking each of fou 
forms of a personality test in three orders of presentation 








Source of 
Variation 


— . 2 
o o 
Freedom Squares Square 


Mean F Fos 

















Tests , 3 21,095.61 7,031.87 189.61** 2.61 3.80 
Groups I 204.64 204.64 Sigs” 8a 6.66 
Orders 2 397-38 198.59 5-36** 3.00 4.62 
Interaction: 
Tests X Groups 3 171.63 57.21 1.54 2.61 3.80 
Interaction: 
Tests X Orders 6 222.19 37-03 1.00 2.10 2.82 
Interaction: 
Groups X Orders 2 49.40 24.70 .67 3.00 4.62 
Interaction: 
Groups X Tests 
X Orders 6 428.52 71.42 1.93 2.10 2.82 
Individual scores within sub- 
classes 1020 37,827.49 37-09 
Total 1043 60,396.86 
* Significant. 


** Highly significant. 


among the individuals taking the four test 
forms, then there is considerably less 
than one chance in a hundred that a 
difference as large as the observed dif- 
ference in the mean score among the 
four tests would be caused by chance 
factors; there is less than one chance in 
a hundred that a difference as large as 
the observed difference in the mean score 
among the three orders would be so 
caused; and there is only from one to 
five chances in a hundred that a dif- 
ference as/large as the observed dif- 
ference in the mean score between the 
two groups would accidentally occur. 
Stated differently: if the individual score 
variation is taken as a valid estimation 
of error variance, it may be stated with 
great confidence that the obtained test 
and order differences, and with reason- 
able confidence that the observed group 
differences, did not arise by chance. 

If, instead of using the variation 
among individual scores as the best esti- 
mate of variance, the standard error of 
estimate is calculated by combining the 





appropriate mean squares of first-order 
interactions, second-order interactions, 
and individual scores, and dividing by 
their combined degrees of freedom, the 
F-values for tests, groups and orders 
respectively become 188.37, 5.56, and 
5-32—which are almost identical with the 
values in Table 2. This analysis of 
variance, evidently, is so stable that the 
F-values maintain their significance even 
when their base is changed—a fact which 
should enhance confidence in the sig 
nificance of the observed mean differ 
ences. 

Another way of interpreting the three 
significant F-values of Table 2 is to say 
that, with the influence of groups and 
orders eliminated, the mean scores of the 
four tests were significantly different. 
With the effect of tests and orders 
eliminated, the group differences wert 
significant. Finally, with effects of tests 
and groups eliminated, order of presente 
tion differences were significant. 

Problem 1b. Did presentation of all 
four test forms in a single battery pro 
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duce significant interaction effects? 

Since the first- and second-order inter- 
action F-values of Table 2 are all low, 
and none of them reaches significance 
at even the 5% level of confidence, there 
is no evidence that significant degrees 
of interaction among tests, groups, or 
orders took place. 

Problem 1c. Were there significant 
differences between scores obtained on: 
The first and second direct tests? The 
first and second indirect tests? The direct 
and indirect tests? 

Table 2 must be analyzed into smaller 
segments to provide more detailed con- 
clusions to several problems of the ex- 
periment. Thus, although the F-value 
of 189.61 for tests shows that the four 
tests differed significantly among them- 
selves, it does not clearly indicate 
whether D-1 mean scores differed from 
those of D-2, I-1 from I-2, nor D-1 and 
D-2 from I-1 and I-2. However, if the 
mean square for tests is appropriately sep- 
arated into the three degrees of freedom 
of which it is comprised, these questions 
may be answered: 

1. Between the mean scores of Tests 
D-1 and D-g, F-values of 2.45 and .oz 
are obtained, respectively, for non-prob- 
lem and problem subjects. Neither of 
these F-ratios is significant at the .os, level 
of confidence. 

2. Between Tests I-1 and I-2 highly 
significant F-ratios of 49.07 and 65.06 
are obtained, respectively, for non-prob- 
lem and problem children, Apparently, 
the two direct tests did not differ sig- 
nificantly from each other, but the two 
indirect ones did. 

3. Between the two direct tests, on the 
one hand, and the two indirect sched- 
ules, on the other hand, F-ratios are 
found to be, for non-problem and prob- 
lem children respectively, 101.48 and 


8.05, both of which are highly signifi- 
cant. Thus, the two direct tests differed 
distinctly from the two indirect forms. 

Problem 1d. Which of the four tests 
significantly differentiated the problem 
from the non-problem groups of sub- 
jects? 

While Table 2 indicates that there 
were significant differences between the 
mean scores of the problem and non- 
problem children, it does not specifi- 
cally show which of the four tests ac- 
counted for these differences. Accord- 
ingly, the data were arranged so that 
analyses of variance could be done by 
individual tests. These anlayses are pre- 
sented in Table 3 

From Table g it may be seen that the 
F-ratios for group differences for Tests 
I-1 (3.08), I-2 (.o2) and D-1 (2.09) are 
not significant. However, that of Test 
D-2 (7.26) is quite significant. In other 
words, on Tests I-1, I-2, and D-1 there 
is a good chance that differences as 
large as the observed differences be- 
tween the mean scores of problem and 
non-problem children could have arisen 
purely by accident. But there is less than 
one chance in a hundred that the ob- 
tained group difference on Test D-2 was 
an accidental one. 

Problem re. Did order of presentation 
of the questions make a significant dif- 
ference on test scores? 

Table 2 indicates that order of presen- 
tation of the questions effected signifi- 
cant differences in mean test scores. 
Table 3 indicates that when problem 
and non-problem groups were considered 
together, significant order differences oc- 
curred with Test I-2, but not with 
Tests I-1, D-1, and D-2. When analyses 
of variance are performed on the mean 
test scores with groups and tests con- 
sidered separately, the F-ratios shown in 
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TABLE 3 


Analysis of variance of mean scores. 221 non-problem and 40 problem children taking each of four 


forms of a personality test in three orders of presentation pe 
: se 


lem a 








Source of D Pe — Mean F Fe F 
. . ¢ “0 
Variation ier Senuiiin Square ‘ 
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Test I-1 
Groups I 213.13 213.13 3-08 x, 6.76 
Orders 2 312.28 156.14 2.26 3.04 4.71 
Interaction: Non- 
Groups X Orders 2 59.65 29.73 43 3-04 4.71 “_ 
Individual scores 255 17,621.94 69.11 Te 
Total 260 18,207.00 Ls 
Me I-2 Te 
£ roups I 9.47 9-47 -02 3.89 6.76 
i Orders 2 495-77 247.83 5.51% 3-04 4.71 Prob 
# Interaction: Te 
j Groups X Orders 2 19.72 9.86 -22 3-04 4.71 Te 
Individual scores 81 Te 
Total 260 11,742.77 ie 
Test D-1 n 
Groups I 43-39 43-39 2.09 3.89 6.76 
Orders 2 1.12 56 -03 3-04 4.71 
Interaction: 
Groups X Orders 2 85.81 42.91 2.16 3-04 4-71 (4.7 
Individual scores 255 5,171.69 20.28 tist! 
Total 4 
D-s 
Test D-2 : 
Groups I 110.29 110.29 7. 26°* 3.89 6.76 on 
é Orders 2 .82 .41 .02 3.04 4-71 alte 
# Interaction: 
5 Groups X Orders 2 73.32 36.66 2.42 3-04 4.71 (4.5 
B Individual scores 255 3,865.26 15.16 is 1 
b “ —_— — 
i Total 260 4,029.69 F 
. ** Highly signifi ny test 
ighly significant. or 
Table 4 are obtained. In these analyses, than on the alternate (14.83) or the nat 
the variance among the mean scores inthe _direct-indirect (12.62) orders of presenta- ore 
three orders (indirect-direct, alternate, tion. This increase in indirect-direct I-29 
' and direct-indirect) are compared with mean score is statistically significant. SCO 
Ee the estimate of error variance obtained 2. Non-problem children taking Test gel 
P by considering the variation among the I-2 also obtained higher scores on the Inc 
re scores of the individuals taking each test indirect-direct (13.33) than on the alter- eve 
ss order. nate (11.81) or the direct-indirect (10.00) nif 
t If Table 4 is read in conjunction with orders. This, too, is a statistically signif- 
H Table 1 the following points may be cant increase. ve 
; noted: 3. Non-problem children taking test tio 
i 1. Non-problem children taking Test D-1 obtained a lower (or better) mean pr 
I-1 obtained a higher (that is, worse) score on the indirect-direct (4.20) than pr 
mean score on the indirect-direct (15.73) the alternate (4.41) or the direct-indirect lon 
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TABLE 4 


Order of presentation F-Ratios (order of pre- 
sentation variance/error variance), 221 non-prob- 
lem and 40 problem children taking each of four 
forms of a presonality test in three 
rs of presentation 








Observed 
Value 


Test F. 06 
of F 


Fin 





Non-Problem 

Children 
Test I-1 
Test I-2 
Test D-1 
Test D-2 


Problem 
Children 
Test I-1 .02 
Test I-2 -98 
Test D-1 -45 
Test D-2 -47 





* Significant. 
** Highly significant. 


(4.77) order. This decrease is not sta- 
tistically significant. 

4. Non-problem children taking test 
D-2 again obtained a lower mean score 
on the indirect-direct (3.75) than on the 
alternate (3.87) or the direct-indirect 
(4.35) order of presentation. This, too, 
is not a significant difference. 

5. Problem children taking the I-1 
test obtained a lower mean score on the 
indirect-direct (16.54) than on the alter- 
nate (17.21) or the direct-indirect (16.85) 
order, But problem children taking Tests 
I-2, D-1, and D-g obtained higher mean 
scores on the indirect-direct than they 
generally did on the alternate and direct- 
indirect orders of presentation. How- 
ever, none of these differences are sig- 
nificant. 

The data of Tables 1 and 4 seem to 
confirm the hypothesis of test sophistica- 
tion. For when the direct questions were 
printed prior to the indirect ones, non- 
problem subjects made significantly 
lower, or better, scores on both indirect 


tests. But when the indirect questions 
were printed first, subjects made some- 
what (though not statistically reliable) 
lower, or better, scores on both direct 
tests. It would seem that presenting the 
direct questions prior to the indirect ones 
enabled non-problem children to dis- 
cover the intent of the latter and answer 
them in a favorable manner. Presenting 
indirect questions prior to direct ones 
also produced a degree of test sophisti- 
cation on the latter, but not enough to 
make the mean score differences statisti- 
cally significant. The fact that, in all 
four tests taken by non-problem chil- 
dren, the mean scores on the alternate 
order of presentation were always in be- 
tween those obtained on the indirect- 
direct and direct-indirect orders lends 
additional weight to the theory that a 
measurable degree of test sophistication 
took place. 

Concerning the behavior problem chil- 
dren, the evidence on order of presenta- 
tion differences is conflicting and baf- 
fling. It is probable that since so few 
problem subjects were employed in each 
sub-grouping, and since the sub-groups 
may have differed widely among them- 
selves, the observed order differences 
here—none of which reaches statistical 
significance—are meaningless. 

Problem 2a. How did the internal 
consistence or reliability of the four 
tests affect the subjects’ scores? 

It is important to know the internal 
consistency or reliability of each of the 
four tests to see if lack of test reliability 
may have accounted for some of the 
observed differences. As Jackson and 
Ferguson (10) have pointed out, what 
does or does not constitute a true test 
reliability is still a matter of controversy. 
Test-retest, split-half, internal consis- 
tency and other methods commonly used 




















Mi © ee Neel 





oe tr te Ry ag ig IR aE, cages Pt tat i = 











GES ptt Me gy eg cee ENT REN 


sae ical hole dim,...o 


for obtaining reliabilities each has its 
advantages and disadvantages. In the 
present study, the Kuder-Richardson (11) 
formula for measuring internal consis- 
tency was employed—because the four 
test forms were administered only once 
and because this formula gives a single 
unique estimate of internal consistency 
or reliability. 

Table 5 presents the internal con- 
sistency quotients obtained by applying 
the Kuder-Richardson formula No. 20 
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TABLE 5 


interaction among tests, groups, an 
assemblies as the error of estimate, nop. 
significant F-ratios were found for test 
(.92), groups (.18) and orders (.22).* |; 
would seem, therefore, that there were 
no significant test, group or order dif. 
ferences in the internal consistency quo. 
tients.. Apparently, the observed mean 
score differences were not attributable to 
differences in the internal consistency of 
the four test forms. 

Problem 2b. Were the obtained dif. 


Internalconsistency or reliability coefficients (Kuder-Richardson). 221 non-problem and 40 problem 


children taking each of four forms of a personality test in three orders of presentation 








Internal Consistency Quotient 
of Non-Problem Children 


Internal Consistency Quotient 
of Problem Children 








Order of Presentation 





Order of Presentation 











Indirect- Alternate Direct- Indirect- Alternate __Direct- 

Direct Indirect Direct Indirect 

(N=69) (N=78) (N=74) (N=13) (N=14) (N = 13) 
I-1 -90 -92 . 86 -76 -QI 93 
I-2 .89 .go .76 -70 -63*** 88 
D-1 .82 -gt 78 -94 .83 75 

D-2 .78 -79 .78 .84 -94 sor 





*** All the correlations in the table are statistically significant from zero at the 1% level of conf- 


dence except the two starred ones. 


to the original score data of the experi- 
ment. It can be seen that while none of 
the correlations in Table 5 is exception- 
ally high, those of the non-problem chil- 
dren are all significantly different from 
zero at the 1% level of confidence, ac- 
cording to Fisher’s (6) test for the signifi- 
cance of correlations. Those of the prob- 
lem children, however, are not as high 
as those of:the non-problem subjects. 
The internal consistency quotients in 
Table 5 may be converted into z-values 
(Fisher’s transformation) and tested for 
significant differences among | them. 
When this was done, and an analysis 
of variance performed upon the z-values 
thus obtained, using the second-order 


ferences in mean scores attributable to 
lack of item stability on the four tests: 

To discover if the tests possessed a 
sufficient degree of item stability to war- 
rant the assumption that the obtained 
test, group, and order mean score dil 
ferences were not merely caused by the 
use of unstable testing instruments, an 
item analysis was done of each of the 
four tests used with non-problem chil: 


* This analysis of variance of the z-values, 4 
well as those made in the next few pages, is 
based on the assumption of equal reliability for 
the original correlations and therefore is not (0 
be viewed as too precise a test of significance of 
test, group, and order differences. While it is the 
best that can be made under the circumstances, 
its results should be taken as approximations 
rather than as exact determinations. 
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dren. A similar analysis was not done for _ biserial correlation was found between 
the problem children’s responses be- total test score and each item response. 
cause the small number of problem The obtained correlations, classified ac- 
children employed in the experiment cording to order of presentation of the 
made it impractical. tests, are shown in Table 6. An exam- 

In the item stability analysis, the ination of Table 6 reveals the following: 


TABLE 6 


Bi-serial correlations between responses to individual test items and total test scores. 
221 non-problem children taking each of four forms of a personality test 
in three different orders of presentation 








Bi-serial Correlation Coefficient Between Item and Total Score 





Test I-i Test I-2 Test D-1 Test D-2 
I-D* ALT D-Ie I-D ALT D-l I-D ALT I-D ALT D-lI 








.30 )=— «. 87 -57 40 .gO .70 (BS)! LG ge 79 SSP ag 
48 .85 .79 588i ge. .58 See a we 
73 (ae .89 6. - 136° 68 -42 alee cere) Bee 
-59 .89 = .88 J5e 5°65 260 A eae ‘ 41 .74 69 
76 -74 #««.81 -53 -78 .07 -37 -55 , -Q2 . 88 -95 
-69 .88 .92 79 s55 81 Jo. 80. «. BS ‘ 48. .2B. saE 
-38 86.80 — . 86 68 ..65 ° .67 (RSET km? .50 .g2 .69 
-53 -49 ~#«2~«. 82 -39 -71 a .70 .94 , .31 .89 -41 
A . 289-3 ee -45 .68 .68 7 ee -95 


. -75 a 
Se ge By 6 785 16g a 88 ; -46 .42 8.89 


28... 63. 63 we eee. Os .: =e — -gI .88 
-43 -QI Ol Hsisy -49 -57 ; . 89 . -45 -41 -32 
-45 -64 «.78 .82 -39 -43 , .60 ‘ -53 .88 .46 
69 .64 .71 -44 .48 .67 ‘ 63 k aI ee 
74 78S) 1.68 gh): Be a : Sr ti .60 .Q1 .88 
OB) ocOR 2) a .88 ‘ Se , 120, -,08. . «68 
+73 -72 ; -42 .48 .82 A . 89 - et .08 .59 
29° Tegel eo. -43 .08~ .58 ; ‘gr. 49 .7O  .63 
73 «7Ty s70:. Ba. .69 ° -45 : 41 -31 -50 
-54 .30 . .78 .16 79 ; -75 : -59 .82 


le to é Soe wear Qt SR . Oo. % -55  -Or 
ae . Sa oe \. <r | Pals ; —_— .58 .60 
ests: : 36 see 461 TeSSo° - ; 92. 43 .28 
od a Y Fe ee A re eS ; cc Sie go ..Ss 
é -§0.' + OZ* .61 87 . F . .48 .67 
Wal- : +35 +73 ; GOs i4he . -52 : .30 tt 
ined . 5 See, ae -92 -92 A : .95 ‘ eee 
t 5. be es a ae” : we. 17 - 6 
dif : Bivvy? bo. , ann. oe : eae .20 =. 68 
+ the . -39° 48%: . —s6 . 40 66 -65 .40 


3, all ‘ -67 .46 . -74  ~««.82 ; . OR: +27 .88 
the 32. <° eet ee 21 _—"~ -42 : .46..° .98 
rd 33. -59 .76. 2 eae a ; ala 80 .92 
chil- 34. -60 .977 : .85 -43 . ‘ -57 ‘ .63 -52 
-57 +35 ‘ +71 -44 . : -94 : +37 -95 
es, a8 . so. ee: 139 = «62 . 61 ‘ -14 .40 
res, is 
ty for Average 58 -71 ‘ -63 -64 P ‘ -79 ‘ .46 <3 
ot to 
ce of * Indirect-direct order of presentation. 
is the > Alternate order of presentation. 
ances, * Direct-indirect order of presentation. 


ations * Blank spaces indicate that there were no responses to one of the categories, all the individuals 
answering the item in one way. 
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1. All the correlations are positive— 
which indicates that every item in the 
four tests contributed something to the 
schedule in which it was used. Most of 
the correlations are above .42, and are 
consequently significantly different from 
zero at the 1% level of confidence. 

2. When the correlations in Table 6 
are transformed into z-values and average 
correlations calculated for each order of 
presentation, the averages are all signifi- 
cantly different from zero at the 1% 
level of confidence. Evidently, then, the 
tests employed in the experiment were 
not composed of haphazardly thrown to- 
gether items, but did possess some degree 
of item stability. 

The biserial correlations in Table 6 
may be transformed back into z-values 
and analyzed for significant differences 
among them. When this was done, using 
the mean square of the individual items 
as the estimate of error variance, analyses 
of variance produced F-ratios of 8.00 
for tests (3 and 420 degrees of freedom) 
and 62.00 for orders (2 and 420 degrees 
of freedom), both of which are highly 
significant. That is to say: there were 
significant differences among both tests 
and orders, thus confirming the test and 
order mean score differences previously 
discovered. 

Problem 3. What was the effect of the 
intelligence quotients of the subjects on 
their personality test scores? 

To uncover the possible effect of in- 
telligence quotients on personality test 
scores, the Pintner Intermediate Test of 
Mental Ability was given to the subjects. 
On this test, the non-problem children 
had an average intelligence quotient of 
109.74 and the problem children of 
100.42. However, the problem children’s 
mean chronological age was 13.82 against 
the non-problem children’s mean of 
12.84. Mean mental ages of the problem 
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children were found to be, for the jp. 
direct-direct, alternate, and  direct.ip. 
direct orders respectively, 13.85, 13.97 
and 13.86; while those of the non-prob. 
lem subjects for the same respectiy 
orders, were 14.11, 14.21, and 193.9, 
When tested by Chi-square analysis 
these were not found to be significantly 
different, since Chi-square equalled on 
.84 for two degrees of freedom. 

The correlations between the intelli 
gence quotients of. the individual chil. 
dren and their scores on the four tests 
are shown in Table 7. Table 7 indicates 
that whereas in many instances the non. 
problem children showed a significant 
negative degree of relationship between 
their intelligence quotients and test 
scores (that is, the higher their intelli 
gence, the lower and more favorable 
their scores), there was only one notable 
instance of this occurring with the prob- 
lem children. 

To test the significance of the group, 
test, and order differences in the correla 
tions listed in Table 7, these were con- 
verted into z-values and an analysis of 
variance performed. Using the mean 
square of the second-order interaction 
(tests X groups X orders) as the estimate 
of error variance, these F-ratios were 
found: for groups, 4.11 (1 and 6 degrees 
of freedom); for tests, 1.55 (3 and 6 
degrees of freedom); and for orders, 2.8 
(2 and 6 degrees of freedom). All these 
F-ratios are non-significant, indicating 
that the observed mean score differences 
among tests, groups, and orders probably 
were not caused: by differential intelli 
gence quotient effects. 

Problem 4. What were the intercor- 
relations between: The direct and in- 
direct tests? The two direct tests? The 
two indirect tests? 

The intercorrelations between the 
scores of non-problem and problem chil 
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he in- TABLE 7 


ect-in- Correlation coefficients between intelligence quotients and scores. 221 non-problem and 40 problem 
13.99 children taking each of four forms of a personality test in three orders of presentation 
“Ji 


P rob Correlation Coefficient of Correlation Coefficient of 
CLIVE Non-Problem Children Problem Children 











13.90, Relationship Order of Presentation Order of Presentation 


alysis, 
canth Indirect- Alter- _Direct- Indirect- Alter- Direct- 
Direct nate Indirect Direct nate Indirect 
1 only (N=69) (N=78) (N=74) (N=13) (N=14) (N=13) 
Intelligence Quotients and I-1 Test 
ntelli- Scores : 04 —.41** 43°? 25 44 09 
| chil Intelligence Quotients and I-2 Test 
{ Scores — .26* — .49** .64** .26 —.22 .66* 
r tests Intelligence Quotients and D-1 Test 
licat Scores k —.44 
. Intelligence Quotients and D-2 Test 
> non- Scores .06 —.14 —.47** , .06 -45 


ificant 








ed .O1 —.31** : -14 -16 


* Significantly different from zero at the 5% level of confidence. 
tween ** Significantly different from zero at the 1% level of confidence. 


L test 

ntelli- dren on all four test forms are presented Tests D-1 and D-2 was reasonably high 

orable in Table 8. From an examination of for the non-problem, but not for the 

otable Table 8 these points may be noted: problem children. 

prob- 1. The two indirect tests, except for 4. The indirect-direct order of presen- 
the indirect-direct order taken by the tation generally resulted in higher test 





sroUup, non-problem children, showed only a_ intercorrelations than did the direct-in- 
yrrela- low degree of intercorrelation. direct order. 

> CON. 2. The degree of correlation between When the correlations in Table 8 were 
sis of the indirect and direct tests was not very converted to z-values and an analysis 
mean high for either the problem or non- of variance done upon them, using the 
action problem subjects. mean square of the second-order inter- 
imate 3. The degree of relationship between action as estimated error variance, these 


were 
TABLE 8 
egrees 


d 6 Intercorrelation coefficients for four personality tests taken by 221 non-problem and 
im 40 problem children in three orders of presentation 





5, 2.84 
these Intercorrelation Coefficient Intercorrelation Coefficients 

cating of Non-Problem Children of Problem Children 

rences Relationship Order of Presentation Order of Presentation 


bably Indirect- Allter- Direct- Indirect- Alter- Direct- 
ntelli- Direct nate Indirect Direct nate Indirect 
(N=69) (N=78) (N=74) (N=13) (N=14) (N=13) 














, Tests I-1 and I-2 pyr? . — .09 -47 —.23 -17 
ere Tests I-1 and D-1 —.o1 i ag?” 54 
din Tests I-1 and D-2 .09 af . 26* .66* 
. The Tests I-2 and D-1 .24* * 22 .66* 
Tests I-2 and D-2 .37°* . -04 .61* 
Tests D-1 and D-2 .83** 79°" .67** .97** 





the 


chil * Significantly different from zero at the 5% level of confidence. 


** Significantly different from zero at the 1% level of confidence. 
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ent orders), Test I-1 included only 12 sig- 
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F-ratios were obtained: relationships, 
4-13 (5 and 10 d.f.); groups, .56 (1 and 10 
d.f.); and orders, 6.95 (2 and 10 d.f.). 
These F-ratios indicate significant order 
differences at the 5% level of confidence; 
but non-significant group differences. 

Problem 5. How significantly did the 
items on the four tests distinguish be- 
tween non-problem and problem sub- 
jects? 

To discover which test items distin- 
guished significantly between problem 
and non-problem groups, an item analy- 
sis was performed, using the Chi-square 
method as modified by Yates (29) for use 
with small cell figures. Specifically, for the 
36 items on each of the tests, the number 
of non-problem and problem subjects 
answering the item positively and nega- 
tively were recorded in two-by-two tables. 
A Chi-square test for homogeneity of 
each two-by-two table was then made. If 
a sufficiently large Chi-square figure 
(representing a probability of .o5 or less) 
was obtained, it was assumed that homo- 
geneity did not exist between the re- 
sponses of the two groups of subjects, 
and the item was considered to distin- 
guish significantly between them. 

Table g lists Chi-square values only 
for those items which differentiated sig- 
nificantly between non-problem and 
problem children on the four tests em- 
ployed in the experiment, classified ac- 
cording to order of presentation. For the 
purposes of readability, all insignificant 
Chi-square values are omitted from the 
table. 

Observation of Table g reveals the 
following: 

1. Not one of the four tests contained 
many items which significantly differ- 
entiated non-problem from problem chil- 
dren. Out of 108 possible items for each 
test (36 test items given in three differ- 
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nificant ones; Test l-2, 4; Test D-1, 19 
and Test D-2, 17. Statistically considered 
this is a meagre showing since about fiye 
items of each test, on a chance basis 
alone, should be expected to show Chi. 
square values as large as those tabulated 
for the .o5 level of confidence. 

2. The indirect tests included fewer 
differentiating items than did the direc 
tests. However, when the number oj 
direct and indirect test discriminations 
were treated by Chi-square (that is, 16 
out of a possible 216 compared to 27 out 
of 216), Chi-square was found to equal 
.00g for one degree of freedom. The dif. 
ference is not significant. 

3. Only in five out of fifteen instances 
were significant direct test items also sig. 
nificant on one or more of the order 
of the indirect tests. On the other hand, 
seven of the ten significant D-1 items 
were also significant on one or more 
orders of the D-2 test. This substantiates 
previous findings that the two direct 
tests differed from the indirect ones but 
were similar to each other. 

4. Some order of presentation differ 
ences are evident in Table g. Test I 
showed only one significant item value 
for its indirect-direct administration; but 
as many as eight on its direct-indirect. 
Test I-2 had too few significant items in 
any presentation order for realistic com- 
parison. Test D-1 displayed four signif 
cant items on the indirect-direct order 
but only one on the direct-indirect. ‘Test 
D-2 also showed a drop from ten signif: 
cant items on the indiréct-direct order 
to two on the direct-indirect. 

5. Analysis of the content of the items 
which were discriminating in Table 9 
shows no prominent patterning. With 
the direct tests, however, there is an 
interesting finding. The subject mater 
of the eight questions which discrimi: 
nated on one or more of the presentation 
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TABLE 9 


Chi-square values ie ability of individual test items to discriminate significantly 
between non-problem and problem groups of children taking each of four 
forms of a personality test in three crders of prescntaticn 








Obtained Chi-square Value 





Test I-1 Test I-2 Test D-1 Test D-2 
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* Indirect-direct order of presentation. 

> Alternate order of presentation. 

° Direct-indirect order of presentation. 

4 Blank spaces represent non-significant Chi-square values; all printed Chi-square values are signifi- 
cant at least at the 5% level of confidence. 


orders of Tests D-1 and D-g was as fol- No. 10 Finds school a hard place to 
lows: get along in. 
No. 12 Finds it hard to speak out in 
No. 5 Finds it hard to do his school class recitations. 
work right. No. 18 Feels that his teachers treat 
No. g Feels other members of family him badly or unfairly. 
are treated better than he. No. 18 Feels tired when he wakes up 
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in the morning. 
No. 26 Dislikes school. 
No. 27 People find fault with him. 


Five of these eight questions are con- 
cerned with school activities; and these 
happen to be the only questions on school 
activities in the entire test. Thus, the 
problem children gave answers which 
discriminated them significantly from 
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responses to the corresponding I-1 item: 
if 20 of the go children answering the 
I-1 item negatively also answered the 
D-1 item negatively; and if 40 of the 4, 
children answering the D-1 question 
positively also answered the correspond. 
ing I-1 question positively; then the two. 
by-two table set up for determining 
possible item identity from this D-1 to 
I-2 item would look like this: 


Test I-1 











Test D-1 





40 10 ° 
Positive on D-1 Positive on D-1 Total Positive 
and and on 

Positive on I-1 | Negative on I-1 D-1 
4 20 24 
Negative on D-1 Negative on D-1 Total Negative 
and and on 
Positive on I-1 Negative on I-1 D-1 


44 | 
Total Positive 
on J-1 





the non-problem children when specifi- 
cally questioned about their school activi- 
ties. 

Problem 6. Did item identity take 
place between the same questions used 
in: Tests D-1 and D-2? Tests D-1 and 
I-1? Tests D-1 and I-2? 

Since Test D-1 was the basic instru- 
ment used, and the other three forms 
were derived by rephrasing its questions, 
it would be desirable to know whether 
item identity occurred between D-1 
questions and corresponding D-2, I-1 and 
I-2 items. To answer this problem, two- 
by-two tables were set up showing how 
the same subjects responded to corres- 
ponding items in Test D-1 and in each 
of the other tests. Thus, if the 74 non- 
problem children taking the direct-in- 
direct order of the four tests gave 50 
positive responses to a given D-1 item 
and 24 negative responses; if these same 
subjects gave 44 positive and go negative 


30 
Total Negative 


be = 
Total Positive 
on I-1 and Negative 








The specific point at issue in this two- 
by-two table is whether, out of the 24 
children who answered the D-1 item 
negatively, the proportion who also an- 
swered the I-1 item negatively (20/24) is 
greater than the proportion, out of the 
50 children who answered the D-1 item 
positively, who also answered the I: 
item negatively (10/50). For if it can be 
shown that a significantly higher pro- 
portion of the negative than the positive 
scorers on a D-1 item also gave negative 
answers to the corresponding I-1 item, 
then presumably a distinct identity ex- 
isted between the D-1 and I-1 responses 
to this item. In other words, if the hy- 
pothesis of item identity is correct, sub- 
jects who admit the possession of symp- 
toms like frequent crying on Test D-1 
will have a pronounced tendency to re- 
port that children who cry frequently 
are not at all queer (Test I-1); that there 
are many such children (Test I-2); and 
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that they are seldom or never free from 
crying (Test D-2). 

Statistically, the two-by-two tables 
used for item identity analysis were 
tested by the Chi-square formula which 
includes Yates’ correction: 


(ad) — (bce) — N/2}*N 
(a+b) (c+d) (a+c) (b+d) 





When this formula is employed with 
the figures in the two-by-two table pre- 
sented above, Chi-square is found to 
equal 38.4 for one degree of freedom; 
and the null-hypothesis—that this is a 
homogeneous distribution and that item 
identity does not take place—may be 
confidently rejected. 

In accordance with this procedure, the 
36 questions on each of the four tests 
were subjected to an analysis to determine 
whether item identity occurred from 
Tests D-1 to D-2, D-1 to I-1, and D-1 to 
I-e. The results of this analysis for the 
non-problem children’s responses are pre- 
sented in Table 10. Only significant Chi- 
square values are listed in the table. The 
total columns were obtained by adding 
independent Chi-squares for the three 
order columns under each test heading 
and using three instead of one degree of 
freedom.* 

Several observations may be made 
from Table 10: 

1. There was considerable item iden- 
tity between non-problem children’s re- 
sponses to Test D-1 and D-2 items. But 
D-1 to I-1 and D-1 to I-2 identities were 
far less than the D-1 to D-2 identities. 

2. There was decidedly more item 
identity from D-1 to I-2 than from D-1 


* Strictly speaking, this is not entirely accu- 
rate, since the use of Yates’ correction, accord- 
ing to Snedecor (21, p. 193) interferes with the ad- 
ditive nature of Chi-square. The figures in the 
total columns of Table 10 may therefore be taken 
only as suggestive. 


to I-1. Children who admitted the pos- 
session of unfavorable personality symp- 
toms were more willing to see many 
other children as possessing these symp- 
toms than to call others who possessed 
them “queer.” 

3- More questions showed item iden- 
tity than were significantly discriminat- 
ing (according to Table g) between 
problem and non-problem children. 

4- The subject matter of the items 
which showed the most consistent iden- 
tity from D-1 to D-g, D-1 to I-1, and 
D-1 to I-2 was as follows: 


No. 4 Feels like running away from 
home. 

No. 13 Feels that his teachers treat 
him badly or unfairly. 

No. 22 Is afraid of high places. 

No. 26 Dislikes school. 

No. 32 Finds it hard to become a 
leader in games with other children. 

No. 36 Is afraid of his father. 


These items covered the whole range 
of questions used in the tests. There 
does not seem to be any consistent pat- 
ternings among them. 

When an item analysis was done with 
the test results of the problem subjects, 
Chi-square values shown in Table 11 
were obtained. This time, however, the 
three orders of presentation for each 
test were combined, since—as the analy- 
sis of variance of mean scores showed— 
there were no significant order differ- 
ences for the problem children. More- 
over, there were too few problem sub- 
jects taking each order to allow for 
effective subdivisional calculations. 

Examination of Table 11 reveals the 
following: 

1. Item identity from Test D-1 to D-2 
was not as prominent as it was with the 
non-problem subjects; but D-1 to I-2 
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TABLE 10 Ne 
5 Chi-square values indicating a significant degree of item identity from the items on one 
+ personality test to the equivalent items on three other personality tests taken 
by 221 non-problem children in three orders of presentation N 
Obtained Chi-square Value N 
Test D-1 to D-2 Test D-1 to I-1 Test D-1 to I-2 N 
FE I-D* Alt.» D-Ie Tot.4 I-D Alt. D-I Tot. I-D Alt. D-I Tot, T 
4 — 
y Beck MMT SST ait? i re A-4 Ts 49 95 
f 2. 3:5  — — —_ — — — —_— — — 16.8 19.1 
4 3. 2319 ee 6.2 21.8 — _ _ —_ _ 5.5°. = ~~ ? 
§ 4- 04:7. Atr 3-4 30-3 g4 = a EI 3, — 19-4 12.6 32.1 Chie 
¥, — _— — — — — a= _ — — ques — ot ite 
6. 122.9 — 9-9 25.1 — -- “= a _ —_ — — test 
\ 7. — — — _— — — — — -— _— —_ = si 
; 8. 4.0 15.1 19.2 — ee 9:7 — g-t 18.4 
I 9- — 36.9 37-9 74.8 ei et gL ae a = 
3 10. 6° 2s = . 40.4 — — — — 0.2; i= _ 9.8 
* by = rae oe .. aid as pres oy rs ah su Ite 
Es 12. — ——- 36.1 30.4 — — _— _ —_ — 40 — 
% 13. 7.0 18.6. — 27.2 6.7 — 4-9 14.5 — 4.2 — _ dante 
3 14 11.2 — ‘36.8 30.4 — 5.8 = 8.4 _ _— — -- 1 
15. = Bias te - — _ — — QAherrg 1.3 : 
‘ 16. —_— 11.0 6.3 18.0 - — —_— — —_ —_ _ : 
: ng.) Aa SS IO OR Pint on OT AA Oe ant ee gg 
‘ 18, 14-1 10.5 28.9 53-4 _ —_ _— —_ 18.2 9s B78: « 30.2 
2 19. 2.5 oo 4-9 23.0 - _ — _— — _ — -- r 
: 20. 7-5 16.0 5§.9 29.4 - _ — —_ — _ — — | 
; | 
‘es 21. 9-4 22.1 =e 31.5 7 gn — a — 3-7 — { 
B 22. §-9 12.5 13.5 31.9 — — 10.5 14.5 -- 3-7 — — i 
23. 996% ...4338.....6.8 52.0 _ - — _ _ _ —_ — 
24. ra 33.8 15.2 27.6 _— 13$.0.- 5.0 20.8 — — _ I 
25. $0 'O.6 oe 1get _ —_ _ — —'' 16.9 — 13.8 I 
26. ag..6 .44.8.. .—.. 43.8 9:4. 11,83 py 2649 2.9... 13.0..4-> 6 I 
a. — -_ - 7 a —_ eat aes — — —_ - 1 
28. ae —_ — — — —_ om 62S SRB 99.3 I 
29. 30-5 20.7 36.5 87.7 37-9 3-8 — + 43-1 Tyee Sh ek no he I 
30. — 48 — 19.1 — — — — — _ — — I 
I 
‘ 25.8 6. 52.8 — — —_ — — I 
32. 15.21 24:7 — 41.0 4.0 i eee — 9-5 — 10.1 2 
att: sae oni oe “ at 3 ‘hes i Laie ™ 
, G.ef 11.0: 19-3 oon — — _— 4,0. @.9...7.8 2 
35- —' 16.9 4.0 20.9 = - — — — 4.2 °>— — 2 
: — 16.3 12.3 28.8 _— 4.2 = 8.6 23.9 BIL7D 1 —— (35.1 , 
2 
* Indirect-direct order of presentation. 2 
» Alternate order of presentation. 2 
© Direct-indirect order of presentation. ‘ 4 
4 Totals are obtained by adding the independent Chi-squares in the three orders and using three 
degrees of freedom. ; ; 
¢ Blank spaces represent non-significant Chi-square values; all printed Chi-square values are signifi- 
cant at least at the 5% level of confidence. 
identity was somewhat greater. As be- tities made by the non-problem and by 
fore, D-1 to I-1 identity was considerably the problem children. 
less than D-1 to I-2 identity. 3- The subject matter of the questions : 
2. When Table 11 is compared to showing the most consistent identity _ 
Table 10 there seems to be only a moder- from Tests D-1 to D-g, I-1, and I-2 was on 
ate connection between the item iden- as follows: Cai 
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No. 10 Finds school a hard place to 
get along in. 

No. 12 Finds it hard to speak out in 
class recitations. 

No. 24 Gets angry. 

No. 26 Dislikes school. 


Three of these four questions concern 


TABLe it 


Chi-square values itidicating a significant degree 

of item identity from the items of one personality 

test to the equivalent items oft three other per- 
sotiality tests taken by problem children 
(Three orders of presentation combined) 








Obtaitied Chi-square value 


Test D-1 Test D-x Test D-1 
to D-2 to I-1 to I-2 
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* Dashes represent non-significant Chi-square 
values; all printed Chi-square values are signifi- 
Cant at least at the 5% level of confidence, 


school activities. This would indicate 
that the item differentiations of Table 
g, which also were heavily loaded with 
school items, may have been caused by 
the problem children’s reacting quite 
strongly to these items, arid admitting 
negative symptoms on them not only on 
the direct but also the indirect ques 
tions. Theit reactivity in this connection 
tay have possibly been the result of 
their being already known as problem 
children, and therefore having little to 
lose by admissions of unfavorable atti- 
tudes towards school activities. 

Problem 7. Did the degree of embar- 
rassment which the subjects felt in at- 
swering the test items affect their re- 
sponses? 

Since analysis has shown that certain 
items on the test forms discriminated 
significantly betweeti non-problem anid 
problem children, and that item identity 
occurred on some of the questions, it 
may be asked whether the discritiitia- 
tions or item identities are related to the 
degree of embarrassment which the sub- 
jects experienced with the various ques- 
tions. To answer this problem, the direct 
questions employed in the experiment 
(that is, Test D-1) were re-submitted, 
three months after the original admini- 
stration, to one hundred of the original 
non-problem subjects. This time, how- 
ever, the questions were given anony- 


. mously, and the children were requested 


to note whether they considered each 
one to be an embarrassing or a non- 
embarrassing one to which to respond. 
The results of this supplementary survey 
of the subjects’ attitudes towards the 
direct questions are shown in Table 12. 

From Table 12 it may be noted that, 
in general, the children found those 
questions embarrassing which asked 
whether other people were critical of 
them, whether they found it hard to get 
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22 ALBERT ELLIS 
TABLE 12 
Percentage of one hundred non-problem school children finding 
personality test items embarrassing 
Percentage of 
Item Subject Matter of Item 5 cara 
embarrassing 
1. Would rather play by self than with other children ag 
2. Finds it hard to make friénds with other children 14 
3. Cries 25** 
4. Feels like running away from home 29** 
5. Finds it hard to do school work right 19** 
6. Is afraid of the dark 10 
7. Feels ill or weak II 
8. Is afraid of being followed when walking on the street II 
9. Feels other members of family are treated better than he 13 
10. Finds school a hard place to get along in II 





14 
12. Finds it hard to speak out in class recitations _ 8 
13. Feels that his teachers treat him badly or unfairly 13 
14. His feelings are easily hurt 26** 


5. Feels that nobody really loves him 


. Feels tired during the daytime 
17. Finds home a hard place to get along in 





18. Feels tired when he wakes up in the morning 4* 


19. Is afraid during a thunder-storm 
. Has headaches 
. Is afraid of high places 


24. Gets angry 
25. Has the habit of twitching his neck or face 


26. Dislikes school 


. Is bothered with pains in some part of his body 


9 
27. People find fault with him 20** 
28. Parents blame or find fault with him 19** 
29. Bites his nails 17** 
30. Is afraid of fire 5* 
31. Has horrible dreams while sleeping , 
32. Finds it hard to become the leader in games with other children aa** 
33. Becomes gloomy or sad 8 
34. Becomesexcited 10 
35. Stutters or stammers when talking et 
36. Is afraid of his father 13 









* The nine questions which were considered embarrassing by the fewest children. 


along with others, and whether they were 
sensitive to criticism. They found rela- 
tively non-embarrassing the questions 
which asked if they had physical ail- 
ments or were afraid of things like fire or 
high places. There are not many sur- 
prises in the tabulated results. 

When the facts of Table 12 are viewed 


** The nine questions which were considered embarrassing by the most children. 


against those of Tables g and 10, 1 
clear-cut results can be discerned. ‘Thus, 
of the items in Table g which signif- 
cantly differentiated non-problem from 
problem children, three of those of Test 
I-1 fell in the most-embarrassing, three 
in the least-embarrassing categories, and 
the other six in-between. Of the four I 
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discriminating items, one was least-em- 
barrassing, and three in-between. Of the 
seventeen D-g discriminating items, one 
was in the least-embarrassing area, five 
in the most embarrassing, and eleven 
in-between. Of the ten discriminating 
D-1 items, two fell in the least-embar- 
rassing category, three in the most-em- 
barrassing and five in-between. There is 
consequently little to indicate that the 
degree of embarrassment which the chil- 
dren said they felt affected the discrim- 
inative power of the test items. 

Comparing the Table 12 values to 
those of the significant item identities 
listed in Table 10, results are similarly 
inconclusive. Thus, of the twenty-seven 
item identities from Test D-1 to D-e, 
eight fell in the most-embarrassing cate- 
gory, six in the least embarrassing, and 
thirteen in-between. Of the. eight item 
identities from Test D-1 to I-1, one was 
most-embarrassing, three least-embarrass- 
ing and four in-between. Of the thirteen 
identities from Test D-1 to I-2, three 
fell in the most-embarrassing, three in 
the least-embarrassing, and seven in the 
in-between categories. It is evident, there- 
fore, that according to the children’s 
own attitudes towards degree of embar- 
rassment experienced in answering the 
direct test items, there was no noticeable 
degree of correspondence between em- 
barrassment or non-embarrassment on 
the one hand and item identity on the 
other. 

Problem 8. How were the responses 
of the normal children affected by the 
subject matter of the items of the four 
tests? 

To discover whether the subject matter 
of the items of the four tests affected 
the children’s responses, the thirty-six 
questions of each form were divided into 
six categories, according to their con- 
tent. These categories were: (1) questions 


relating to home adjustment; (2) to 
social adjustment; (3) to school adjust- 
ment; (4) to fears; (5) to sensitivity and 
excitability; and (6) to psychosomatic 
symptoms. Listed in Table 13 are the 
six categories, with the percentages of 
non-problem children responding nega- 
tively or unfavorably to each item. The 
problem subjects are not included since 
the small number of cases make their 
percentage figures meaningless. 

Table 13 shows distinct differences 
among the average percentages of sub- 
jects responding negatively to the ques- 
tions in each of the six subject cate- 
gories. When tested with analysis of 
variance, these differences are significant 
for Tests I-1, I-2, and D-1, but not for 
D-2. None of the order differences are 
significant. 

If the percentages in Table 13 are 
averaged by orders (since the order dif- 
ferences are not significantly different) 
the summary information shown in 
Table 13a is obtained. From Tables 13 
and 13a it may be seen that: 

1. On all four tests the subjects were 
least willing to respond negatively to the 
home adjustment and social adjustment 
questions and most willing to respond 
unfavorably to the psychosomatic and 
sensitivity and excitability questions. 

2. There was some degree of corre- 
spondence between the manner in which 
the children responded to the six cate- 
gories of questions on all four tests, but 
there was also some degree of discrep- 
ancy. 

Additional information may be ob- 
tained by noting the nine most-checked 
and the nine least-checked items for each 
of the four tests, This information is 
shown in Table 14. From Table 14 it 
may be seen that on several of the ques- 
tions—especially those concerned with 
the children’s running away from home, 
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TABLE 13 


Percentages of 221 non-problem children giving unfavorable responses to the items of four 
personality tests taken in three orders of presentation 








Percentage of Children Giving Unfavorable Rerrcrse 





Subject Matter of Item Test I-r 


Test I-2 Test D-1 Test D-2 





Altb D-Ie I-D 


Alt. I-D Alt. D-I I-D_ Alt. 





Home Adjustment: 
Feels like running from home 17 16 
Feels family treated better 43 19 
Finds home a hard place 16 3 
Feels parents blame him 20 16 
Is afraid of father 30 21 





AVERAGE 25 15 





Social Adjustment: 
Plays by himself 39 
Finds it hard to make friends 31 
Other boys won't play 39 
Feels nobody loves him 13 
Day-dreams often 3 35 
Hard to become a leader 44 





AVERAGE 





School Adjustment: 
Hard to do school work 
Finds school a hard place 
Hard to speak in class 
Teachers treat him badly 
Dislikes school 





AVERAGE 





Fears: 
Is afraid of the dark 
Afraid of being followed 
Afraid during thunder-storm 
Afraid of high places 
Afraid of fire 
Has horrible dreams 





AVERAGE 








Sensitivity & Excitability: 
Cries easily 
Feelings easily hurt 
Gets an 
People find fault with him 
Feels gloomy or sad 
mes excited 


12 
15 
23 
7 
3 
27 





AVERAGE 


15 





Psychosomatic Symptoms: 

Feels ill or weak 

Tired during the daytime 54 
Tired when waking 80 
Has headaches 66 
Pains in body qt 
Twitches neck or face 27 
Bites nails 54 
Stutters or stammers 67 


3 

8 

37 39 
22 17 
13 5 
I 17 
33 29 
I 7 


wn 
SOWETO D 


Nn 





AVERAGE 60 


: | 


14 16 





a—lIndirect-Direct Order b—Alternate Order 


c—Direct-Indirect Order. 





finding it hard to make friends, tired- 
ness when waking in the morning, and 
biting their nails—there was a definite 
tendency for the subjects to give the 
same kind of response to all four tests. 
But on a few of the questions—specifi- 
cally, those concerned with the subjects’ 
feeling their parents blamed them, being 


afraid of being followed, feeling gloomy 
or sad, and feeling ill or weak—they 
freely gave unfavorable responses to one 
form of the test but rarely gave them 
to another form. 

Table 14 confirms the information of 
Tables 13 and 13a: to wit, that non- 
problem subjects, on all four tests, ad- 
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TABLE 13a 


Average percentages of non-problem children 
of questions on four personality tests ( 


& 


ving unfavorable responses to six categories 
ree orders of presentation combined) 








Average Percentage Giving Unfavorable Response 


Category of Questions 





Test D-: 


Test D-2 Test I-2 Test I-2 





Home Adjustment 

Fears 

Social Adjustment 

School Adjustment 
Sensitivity and Excitability 
Psychosomatic Symptoms 


26. 
39. 
31. 
39- 
37- 
55: 


9-3 
10.3 
10.0 
10.3 
12.7 
11.7 


15.3 
37-3 
20.0 
37-3 
41.7 
35-0 





mitted with relative rarity that they 
were unadjusted to their home and that 
they were socially inadequate; but more 
freely admitted that they were sensitive 
and excitable and that they possessed 
psychosomatic symptoms. This pattern 
of response may be attributable to the 
fact that (1) the children really did possess 
fewer unfavorable home and.social symp- 
toms than sensitivity and psychosomatic 
ones; or (2) they felt less self-criticism in 
admitting the latter than the former 
kind of symptoms. To check this second 
possibility, the information in Table 12, 
showing the degree of embarrassment 
felt by non-problem children in answer- 
ing the direct items, may be compared 
with the data of Table 14. It is then 
found that of the nine questions con- 
sidered most-embarrassing by the chil- 
dren, three were among the least-checked 
items in one or more of the tests, four 
were among the most-checked, one was 
least-checked in two tests and most- 
checked in one, and one was neither 
among the most- or least-checked items. 
Of the nine items considered least-em- 
barrassing by the children, five were 
among the most-checked items on one or 
more tests, and four were among the 
least-checked. Apparently, there was not 
a high degree of correspondence between 
the children’s finding specific items em- 
barrassing, on the one hand, and giving 


unfavorable answers to them on the 
other hand. The observed differences 
among the six categories of questions 
may therefore be attributable to actual 
differences in the subjects’ possession of 
the various traits rather than the result 
of their responding more unfavorably to 
the less self-critical items. Certainly, it 
is easy to believe, as the data of Table 14 
indicate, that non-problem children less 
often feel like running away from home 
and less often find it hard to make 
friends with other children than they 
feel tired when waking in the morning 
and than they bite their nails. 

‘Problem 9. What was the effect of 
the children’s answering certain direct 
and indirect questions out of context as 
against answering identical questions as 
part of the regular test battery? 

The test booklet used in the investiga- 
tion contained three sample questions on 
their covers. The sample D-1 question 
was: “I get angry VERY OFTEN 
PRETTY OFTEN SELDOM NEVER.” 
The sample I-1 quesion was: “Children 
who often day-dream are VERY QUEER 
PRETTY QUEER A LITTLE 
QUEER NOT AT ALL QUEER.” 
The sample I-2 question was: “VERY 
MANY PRETTY MANY FEW NO 
children are afraid of high places.” No 
sample D-2 question was employed, since 
its form was similar to that of the D-1 
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TABLE 14 


The nine most frequently checked and nine least frequently checked items on four 
personality tests taken by 221 non-problem children (orders combined) 








Most-Checked 





Subject Matter of Item 


Least-Checked 





—. 


Test Test 





J-z 





——$—$—$_—. 


I-2 D-1 D-2 I-t I-2 D-1 D-, 





Home Adjustment: 


Feels like running away from home 
Feels family treated better than he is 
Finds home a hard place 

Feels parents often blame him 

Is afraid of his father 





Social Adjustment: 


Plays by himself 

Finds it hard to make friends 
Other boys refuse to play with him 
Feels that nobody loves him 

Day dreams often 

Finds it hard to become a leader 





School Adjustment: 
Finds it hard to do school work 
Finds school a hard place 
Finds it hard to speak in class 
Feels teachers treat him badly 
Dislikes school 





Fears: 
Is afraid of the dark 
Is afraid of being followed when walking 
Is afraid during a thunder-storm 
Is afraid of high places 
Is afraid of fire 
Has horrible dreams 





Sensitivity and Excitability: 
Cries easily 
Feelings are easily hurt 
Often gets an: 
Feels people find fault with him 
Feels gloomy or sad 
Becomes excited 





Psychosomatic Symptoms: 
eels ill or weak 
Feels tired during the daytime 
Is tired when waking in the morning 
Has headaches 
Has pains in his body 
Twitches his neck or face 
Bites his nails 
Stammers or stutters 





questions. The children were not spe- 
cifically requested to answer these sample 
questions, but 101 of the non-problem 
children voluntarily did so, presumably 
as aids to themselves. Accordingly, since 
the sample questions were also contained 


in the regular context of the tests, the 
two responses of these 101 subjects to 
each of the three sample questions were 
compared to see if the contextual one 
was answered in (1) a more favorable, 
(2) a less favorable, or (3) an unchanged 
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manner from the non-contextual re- 
sponses. The results of the comparisons 
are shown in Table 15. 

From Table 15 the following points 
may be noted: 

1. Almost one-fourth .of the children 
answering the sample, non-contextual 
direct question changed their contextual 
response to a more favorable one. At the 
same time, only 4% changed their re- 
sponse less favorably. If it is assumed 
that the non-contextual answers were the 
more truthful ones—since they were pre- 
sumably given when the children were 
caught off their guard, and given to ques- 
tions on which they felt they would not 
be marked—there is some evidence that 
sophistication occurred as a result of the 
test situation. And since the particular 
direct questioned involved here—that 
asking whether the children often got 


angry—was one which, according to 
Tables 12 and 14, the subjects freely 
responded to unfavorably, and accord- 
ing to Table 13 did not deem partic- 
ularly embarrassing, there is some rea- 
son to believe that the test sophistication 
shown here also would prevail for most 
other direct questions. 

2. The direction of the changes made 
in the I-e sample question were similar 
to those of the D-1 question, but not so 
pronounced as to evidence of test sophis- 
tication. Some 18% of the children 
changed their I-2 response in a more 
favorable direction; but 10% also 
changed it in a less favorable manner. 
There seems to be nothing conclusive 
about these changes. i ee 

g. The Test I-1 sample question pro- 
duced results opposite to those of the 
other two questions. This time, the 


TABLE 15 


Changes made by ror non-problem children in answering certain personality test 
questions first out of context and then as part of a regular test battery 








Number and percentage of children making change 





Indirect- 
Direct 
Order 


N % 


Direction of Change 
in Response 


Direct- E 
Alternate Indirect Combined 


Order Grits Orders 
N % N ‘ N % 





D-1 Question (gets angry) 
More favorable 
Less favorable 
Unchanged 
Total 
I-1 Question (day-dreams often) 
More favorable 
Less favorable 
Unchanged 
Total 
1-2 Question (is afraid of high places) 
More favorable 
Less favorable 
Unchanged 


Total 
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children changed their answers to more 
favorable responses 17% of the time and 
to less favorable ones 26% of the time. 
All told, no less than 43% of the sub- 
jects changed their contextual response 
to this item. Several possible explana- 
tions for this unusual response pattern 
present themselves: (1) the children never 
quite figured out what were the favor- 
able answers to this question, and were 
therefore as willing to change their 
answers one way as the other; (2) what 
the scoring system assumed to be “favor- 
able” answers to this test did not prove 
to be so empirically; (3) the subject 
matter of the question—that asking 
about the children’s day-dreams—was 
especially conducive to unstable re- 
sponses. Since none of these explana- 
tions, individually considered, seems 
conclusive, it is possible that a combina- 
tion of them accounted for the observed 
changes. 

Because of the differences in subject 
matter of the three sample questions, it 
is not possible directly to compare the 
changes made on one to those made on 
the other two. However, since the three 
are fairly equivalent in the sense that 
none of them was found to be very em- 
barrassing by the children (Table 12) 
and all tended to be among the most- 
checked responses (Table 14), there’ are 
some indications here that, given the 
scoring procedures used in this investi- 
gation, the D-1 question was somewhat 
transparent to the subjects, the I-2 ques- 
tion less transparent, and the I-1 ques- 
tion non-transparent. This agrees with 
the item identity findings, which showed 
that the I-2 questions were more closely 
related to the D-1 items than were the 
I-1 questions. 


RESUME OF THE FINDINGS 
The major findings of the experiment, 


classified according to the main variable 
involved in it, are briefly listed beloy, 


Tests 


1. The mean scores of the direct tests 
(D-1 and D-2) do not differ significantly, 

2. The average scores of the indirect 
tests (I-1 and I-g) are significantly differ. 
ent. 

3. The mean-scores of the direct 
tests (D-1 and D-g) differ significantly 
from those of both indirect tests (I-1 
and I-g). 

4. The observed mean score differ- 
ences are not attributable to lack of in- 
ternal consistency or lack of item stabil: 
ity of the tests. 

5. The intercorrelations between Tests 
D-:1 and D-g are reasonably high; be 
tween I-1 and I-2, fairly low; and between 
the direct and indirect tests, not too 
high. 

6. Many item identities exist between 
Test D-1 and D-2; a good many between 
D-1 and I-2; but relatively few between 
D-1 and I-1. 

7. The subjects give significantly more 
unfavorable responses to certain kinds 
of questions—e.g., psychosomatic ones- 
than they do to other kinds—e.g., home 
adjustment or social adjustment items. 

8. Subjects taking specified test items 
as part of the regular test battery respond 
differenly than when taking identical 
items out of test context. 

g. The degree of embarrassment the 
subjects say they experience in answering 
the direct questions does\ not affect the 
power of the items to discriminate be- 
tween the two groups or the item identi- 
ties of the questions. 


Groups 
10. All the obtained group differences 
are to be cautiously viewed, since the 
problem subjects are few in number and 
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are probably non-homogeneously sub- 
grouped. 

11. Significant differences between 
problem and non-problem subjects’ 
average scores are evident only for Test 
D-2, and not for Tests D-1, I-1, or I-2. 

ig. Few items discriminate significant- 
ly between problem and non-problem 
groups on any of the four tests, though 
the direct tests contain more item dis- 
criminations than do the indirect. Dif- 
ferent specific items prove discriminating 
on the direct and the indirect tests. 

ig. The items which discriminate 
significantly between problem and non- 
problem groups are largely concerned 
with school activities. 


Orders of Presentation 
14. The order of presentation of the 


direct and the indirect test questions has 
a significant effect upon: 


. Average test scores. 

. Test score intercorrelations. 

. Item discriminations between 
problem and _ non-problem 
groups. 

. Biserial correlations between 
total test and item scores. 

. Percentages of subjects respond- 
ing unfavorably to different 
kinds of questions. 


Interactions 


15. None of the first-order or second- 
order interactions among tests, groups, 
or orders of presentation are statistically 
significant. 
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CHAPTER IV 


DISCUSSION 


INCE Statistically significant differ- 
S ences were found for tests, orders of 
presentation, and groups used in the 
experiment, it may now be asked why 
they occurred and what they mean. 


INTERPRETATION OF THE TEST 
DIFFERENCES 
The indirect tests employed in the 
experiment differed significantly from 
each other as well as from the direct 
tests; and these differences were main- 


tained for both problem and non-prob- 


lem subjects. They may be attributable 
to differences between the indirect and 
direct tests because of (1) format or (2) 
substance. 

It seems unlikely that the test differ- 
ences were the result of format differ- 
ences because: (1) deliberate change in 
phrasing, but avoidance of change in 
meaning, did not bring about signifi- 
cant differences between Tests D-1 and 
D-2. (2) The formats of Tests I-1 and 
I-g differed from each other; yet, both 
I-1 and I-2 mean scores were signifi- 
cantly different from D-1 and D-2 scores. 
(3) Changing the order of presentation 
of the tests, which somewhat changed 
their appearance or format, had no 
significant effect on the distinct differ- 
ences found between the direct and in- 
direct test forms. 

Since there is little reason to believe 
that the format of the tests was re- 
sponsible for the observed mean score 
differences, these would seem to be at- 
tributable to differences in test sub- 
stance. 


INTERPRETATION OF THE GROUP 
DIFFERENCES 


It was found, by statistical analysis, 
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that the problem children reacted differ. 
ently than the non-problem subjects to 
Test D-2; but that there were no signif. 
cant differences between the mean 
scores of the two groups on Tests D.-i, 
I-1, and I-2. The most plausible reasons 
for these diverse group differences seem 
to be as follows: 

1. Tests D-2 and 1-1, which most suc. 
cessfully differentiated the two groups 
(the latter displaying differences which 
were almost significant), seem to have 
been the tests with which the subjects 
had the greatest difficult determining 
the “good” or “correct” answers. Some 
of the D-2 questions were awkwardly 
worded and hence liable to misinter- 
pretation; while few of the I-1 ques 
tions—as the item identity data revealed 
—could be readily identified by the sub- 
jects with their D-1 analogues. Conse- 
quently, the problem children, who were 
found to have lower intelligence quo- 
tients than the non-problem subjects, 
may have answered the D-2 and the I: 
items less sophisticatedly than did the lat- 
ter group: thereby producing the ob- 
served differences. 

2. The group differences could have 
been caused by lack of homogeneity 
among the subgroups of problem chil: 
dren used in the experiment. These 
problem subjects, not being the most 
acute and segregable kind of cases, neces- 
sarily consisted of a heterogeneous popu- 
lation which probably included several 
clinical subgroups. From such a hetero- 
geneous grouping, a sample as small 
as that employed in the experiment 
could easily have varied significantly 
from one order of presentation to an- 
other, thus causing some inconsistencies 
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in the findings. Observation of Table 1 
(p. 7) tends to bear out this possibility 
by showing that while the order of pre- 
sentation scores for the non-problem 
children ran in a consistent direction 
for all tests, those for the problem chil- 
dren were either inconsistent or dispro- 
portional from, order to order. That is 
why sophistication effects found for the 
non-problem group were not also found 
for the problem subjects. The internal 
consistency coefficients (Table 5, p. 12) 
also show evidence of inconsistency in 
the problem children’s responses, and 
this inconsistency again could have been 
caused by differences within the sub- 
groups of problem subjects. Since, 
throughout the entire experiment, the 
non-problem children, whose sub-groups 
always included approximately seventy 
cases, showed fairly consistent findings, 
and the problem children, whose sub- 
groups included approximately thirteen 
cases, often showed inconsistent and con- 
tradictory findings, there must be a 
strong suspicion that the obtained differ- 
ences between the mean scores of the 
two groups resulted from unusual hetero- 
geneity in the makeup of the problem 
subgroups. Consequently, not too much 
emphasis may be placed on the observed 
group differences until they are verified 
by further experimentation. The ob- 
tained test and order of presentation 
differences, on the other hand, stand up 
much better under statistical analysis. 


INTERPRETATION OF THE ORDER OF 
PRESENTATION DIFFERENCES 

The obtained mean score differences 
on the three orders of presentation used 
in the experiment seem to be clearly ex- 
plicable in terms of test sophistication. 
Bolstering the argument that one pre- 
sentation order enabled the subjects to 
become more sophisticated on the tests 


than did the other orders are the follow- 
ing points: 

1. The hypothesis of test sophistica- 
tion would lead to the expectation that 
different orders of presentation would 
produce greater differences on the in- 
direct than on the direct tests, since the 
latters’ questions were so openly stated 
as to reveal their intent no matter what 
presentation order was employed. This 
was precisely what occurred, with order 
differences being significant only on 
Tests I-1 and I-2, and not on D-1 or 
D-2. 

2. If sophistication effects existed, they 
should have been revealed by the sub- 
jects’ receiving higher (or less favorable) 
scores on the indirect tests when they 
answered all the indirect questions be- 
fore the directly stated ones, than when 
they first answered the direct and then 
the indirect questions. This is exactly 
what the experimental data reveals them 
to have done, 

g. If the sophistication hypothesis 
were tenable, it would be expected that 
subjects taking the alternate order of 
presentation would tend to receive mean 
test scores in-between those taking the 
indirect-direct and direct-indirect orders. 
This, again, was revealed by the data. 

4. The sophistication _ hypothesis 
would leave the probability of some 
degree of sophistication taking place on 
the direct tests, since subjects answering 
these tests last (that is, taking the indi- 
rect-direct order) might be made some- 
what test-wise. The obtained scores 
show that some (though a statistically 
non-significant) degree of sophistication 
did occurr on the direct tests, and that 
it was precisely in the expected direc- 
tion. 

On all counts, therefore, the hypothesis 
seems to be upheld; and it would defi- 
nitely appear that the observed differ- 
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ences produced by the orders of pres- vious implications which go with it, may 
entation were attributable to test sophis- be said to be the most important di. 
tication. This finding, and the ob- covery of the investigation. 





CHAPTER V 


SUMMARY AND CONCLUSIONS 


HE PURPOSE of the present experi- 

ment was to study the effects of 
using direct and indirect phrasing in 
personality questionnaires. The specific 
hypothesis to be tested was that if conven- 
tional personality test schedules employ- 
ing direct questions, as well as indirect 
test forms derived from the direct items, 
are given in a single battery to a group 
of behavior problem and a group of non- 
problem school children: (a) the indirect 
test scores will be significantly different 
from the direct scores; (b) groups of be- 
havior problem and non-problem chil- 
dren will react differently to the indirect 
and direct tests; and (c) presentation of 
the questions in three different orders— 
an indirect-direct, an alternate, and a 
direct-indirect order—will have a distinct 
effect on the degree of test sophistication 
shown by the children in their responses. 


SUMMARY OF THE EXPERIMENT 


In order to test the experimental 
hypothesis, these materials, subjects, and 
experimental design were employed: 

Materials. A basic personality test 
form of 36 items was constructed by tak- 
ing the most frequently used questions 
on eight standard personality question- 
naires for children. Two direct forms of 
this test were made: (1) with all 36 ques- 
tions asking about a negative symptom 
(e.g., “I cry VERY OFTEN PRETTY 
OFTEN SELDOM NEVER.”); (2) 
with all 36 questions asking about a posi- 
tive symptom (e.g., “I am free from cry- 
ing VERY OFTEN PRETTY OFTEN 
SELDOM NEVER.”). Two indirect 
forms of the same 36 questions were 
constructed: (1) with all questions ask- 
ing how queer the respondents con- 
sidered other children who possessed 


negative symptoms (e.g., “Children who 
often cry are VERY QUEER PRETTY 
QUEER A LITTLE QUEER NOT 
AT ALL QUEER.”); (2) with all ques- 
tions asking the respondents how many 
other children there were who possessed 
negative symptoms (e.g., “VERY MANY 
PRETTY MANY FEW NO children 
often cry.”). In addition, the Pintner 
Intermediate Test of Mental Ability was 
administered to all subjects. 

Subjects: The experimental groups 
consisted of forty behavior problem boys 
and 221 non-problem boys from regular 
seventh and eighth grade classes of three 
New York City junior high schools. 

Experimental Design. The four test 
forms were bound together in sixteen- 
page booklets, each page containing nine 
questions from one of the tests. The 
booklets were bound so that the tests 
could be administered, in a single setting, 
in three different orders of presentation: 
(1) an indirect-direct order—all eight 
pages of the indirect questions followed 
by the eight pages of the direct ques- 
tions; (2) an alternate order—a page of 
direct questions alternated with a page 
of indirect ones; and (g) a direct-indirect 
order—all eight pages of the direct ques- 
tions followed by the eight pages of the 
indirect ones. The booklets were dis- 
tributed so that one-third of the problem 
and one-third of the non-problem sub- 
jects. received each order of presenta- 
tion. Problem and non-problem children 
were tested in their regular home-room 
classes, with no segregation of the groups 
taking place. 


CONCLUSIONS AND IMPLICATIONS 


The results of the experiment sub- 
stantially support its hypothesis, the 
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main findings being as follows: 

1. The mean scores of the direct tests 
do not differ significantly from each 
other; but those of the indirect tests 
do. The mean scores of the two direct 
tests differ significantly from those of 
both indirect tests. 

2. Intercorrelations between the two 
direct tests are reasonably high; and 
many item identities exist between these 
two tests. Intercorrelations between the 
two indirect tests are fairly low. Inter- 
correlations between the direct and in- 
direct tests are not too high; and item 
identities from the direct to the indirect 
tests are not too many. One indirect 
test (I-2) is much more closely related to 
the direct tests than is the other in- 
direct test (I-1). 

3. The subjects give significantly more 
unfavorable responses to certain kinds 
of questions, especially psychosomatic 
ones, than they do-to other kinds, 


especially home and social adjustment 
items. 


4. The subjects react quite differently 
to questions taken out of the test context 
than they do to those taken as part of 
the regular test battery. 

5. Significant differences between 
problem and non-problem subjects’ 
mean scores are evident only for one of 
the direct tests (D-2) and not for the 
other direct test or for either of the 
direct tests. Observed differences be- 
tween problem and non-problem groups 
of subjects are probably atributable to 
the small number and non-homogeneous 
subgrouping of the problem children 
and are therefore to be 
viewed and interpreted. 

6. Few items discriminate significantly 
between the problem and non-problem 
groups on any of the four tests, though 
the direct tests contain more item dis- 
criminations than do the indirect. Dif- 


cautiously 


ferent specific items prove discriminating 
on the direct and indirect tests. The 
items which discriminate best between 
problem and non-problem groups are 
largely concerned with school activities, 

7. The order of presentation of the 
direct and the indirect test questions 
has a significant effect upon average test 
scores, test score intercorrelations, item 
discriminations between problem and 
non-problem groups, biserial correla. 
tions between total test and item scores, 
and percentages of subjects responding 
unfavorably to different kinds of ques. 
tions. 

The main implications to be drawn 
from the experiment are presented be- 
low. In making them, it must be em. 
phasized that the present study, being 
based on the conventional types of ques. 
tionnaires which are now widely used, 
is doubtlessly limited by some of the 
defects which are inherent in such tests. 
A particular limitation of the study is 
its use, for its problem subjects, of a 
smaller group of children than would 
have been statistically desirable. Con- 
sequently, the following conclusions 
and implications must be taken as tenta- 
tive, and as requiring for their final con- 
firmation considerable further experi- 
mentation, { 

1. The order of presentation of indirect 
and direct personality questions ad- 
ministered in a single test battery may 
decisively effect test results. If test-makers 
are to construct instruments that will 
encourage the least sophisticated, most 
truthful responses by their subjects, they 
must make certain that all indirect 
items are printed before the direct ones. 

2. Phrasing is an important considera- 
tion in the construction of personality 
test items. Each mode of indirectly 
phrasing a schedule of questions may re- 
sult in the creation of an instrument 
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significantly different from that of other 
indirectly phrased questionnaires. 

3. Indirect personality tests seem to 
differentiate problem from non-problem 
groups of children no better than do 
direct test forms. Certain specific ques- 
tions, however, are more discriminative 


when indirectly ‘than directly phrased, 


so that optimum results may possibly be 
obtained by combining both kinds of 
questions in a single test. 


4. The kinds of questions which ap- 
pear best to distinguish problem from 
non-problem groups of children are 
those specifically concerned with school 
activities. b 

5. Much more experimental work 
must be done with personality question- 
naires before they can be expected to be 
satisfactory instruments for the diagnosis 
of behavior problem difficulties. 
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APPENDIX 
THE TESTS AND INSTRUCTIONS EMPLOYED IN THE EXPERIMENT 


Instructions 
(To be read aloud to the Class) 


Do not open your booklet until you are given the signal to do so, We would like your help 
in finding out what school children think about themselves and what they think about other 
children. Please answer the questions as honestly as you can, so that we may find out what 
school children, including yourself, are really like. Of course, this is not a test. These questions 
have nothing to do with your school work. Your teacher will not see them. You will not be 
marked on them. 

Look at the top of the page in front of you. Fill it out now. Write in your name. Write 
today’s date. Write down how old you were at your last birthday. Circle whether you are a 
boy or a girl. Put down your official class. Write the number of this school. 

Do not open this booklet until you are given the signal to go. When you do open it, you 
will find that it contains three different kinds of questions. Samples of each one of these 
three kinds of questions are printed on the page before you. You may now look at these 
sample questions. First look at Question 1. Read it to yourself as I read it to you. It says: | 
get angry VERY OFTEN PRETTY OFTEN SELDOM NEVER.” This question asks 
something about yourself. Answer it truthfully about yourself. Some of you may get angry 
VERY OFTEN, some of you PRETTY OFTEN, some SELDOM, and some of you NEVER. 
One of these descriptions certainly fits you. So draw a circle around the one which does 
describe how angry you get. Remember: only draw a circle around ONE answer for each 
question. 

Now look at Question 2. This question reads: “VERY MANY PRETTY MANY FEW 
NO children are afraid of high places.” It asks about other children whom you know. We 
want your opinion about how many of these other children are afraid of high places. Again, 
circle only one of the four answers to this kind of question. 

Now look at Question 3, which reads: “Children who often day-dream are VERY QUEER 
PRETTY QUEER A LITTLE QUEER NOT AT ALL QUEER.” This question, again, 
asks about other children. We want your opinion about children who often day-dream, and 
want you to let us know if you personally think they are VERY QUEER, A LITTLE 
QUEER, or NOT AT ALL QUEER. This is not a test, remember, and you will not be 
marked on it. 

You can see, then, that there are three different kinds of things you are to do in this 
booklet. One kind of question asks about yourself, and the other two kinds ask your opinions 
about other children. In all three kinds of questions, though, you are to draw a circle around 
only ONE of the four possible answers. Carefully read each question and answer it as 
honestly as you can. Do not skip any of them. The booklet has sixteen pages in it, but it 
should not take too long to answer all its questions. There is no speed limit, but work as 
fast as you can. 

Your truthful answers on this booklet will be a great help to us in finding out what school 
children are really like, so be sure to answer every single question carefully. ‘This has nothing 
to do with your school work, and your teacher will not even look at your papers. 

Now open the booklet and make sure that there are exactly sixteen white pages of ques- 
tions. If there are, go right ahead and answer all the questions. If there are not, ask your 
teacher for anew booklet. 
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Test D-1 Questions 


I would rather play by myself than play 
with other children VERY OFTEN PRETTY OFTEN SELDOM NEVER 
I find it hard to make friends with other 


children VERY OFTEN PRETTY OFTEN SELDOM NEVER 





I cry 


| feel like running away from home 
| find it hard to do my school work right 
| am afraid of the dark 


VERY OFTEN 
VERY OFTEN 
VERY OFTEN 
VERY OFTEN 
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PRETTY OFTEN 
PRETTY OFTEN 
PRETTY OFTEN 
PRETTY OFTEN 
PRETTY OFTEN 


| feel ill or weak VERY OFTEN 
| am afraid of being followed when I am 

walking on the street VERY OFTEN PRETTY OFTEN 
Other members of my family are treated 

better than I am treated VERY OFTEN PRETTY OFTEN 


| find school a hard place to get along in VERY OFTEN PRETTY OFTEN 
Other children refuse to play with me VERY OFTEN PRETTY OFTEN 
I find it hard to speak out in class 

recitations VERY OFTEN PRETTY OFTEN 
] feel that my teachers treat me badly or 

unfairly VERY OFTEN PRETTY OFTEN 
My feelings are easily hurt VERY OFTEN PRETTY OFTEN 
] fee] that nobody really loves me VERY OFTEN PRETTY OFTEN 
I feel tired during the daytime VERY OFTEN PRETTY OFTEN 
I find home a hard place to get along in VERY OFTEN PRETTY OFTEN 
I feel tired when I wake up in the 

morning VERY OFTEN PRETTY OFTEN 


I am afraid during a thunderstorm VERY OFTEN PRETTY OFTEN 
I have headaches VERY OFTEN PRETTY OFTEN 
I am bothered with pains in some part 


of my body 


I am afraid of high places 


I day-dream 
I get angry 


VERY OFTEN 
VERY OFTEN 
VERY OFTEN 
VERY OFTEN 


PRETTY OFTEN 
PRETTY OFTEN 
PRETTY OFTEN 
PRETTY OFTEN 


I have the habit of twitching my neck 

or face VERY OFTEN PRETTY OFTEN 
I dislike school VERY OFTEN PRETTY OFTEN 
People find fault with me VERY OFTEN PRETTY OFTEN 


My parents blame me or find fault 

with me VERY OFTEN PRETTY OFTEN 
I bite my nails VERY OFTEN PRETTY OFTEN 
I am afraid of fire VERY OFTEN PRETTY OFTEN 
I have horrible dreams while sleeping VERY OFTEN PRETTY OFTEN 
I find it hard to become a leader in 

games with other children VERY OFTEN PRETTY OFTEN 
I become gloomy or sad VERY OFTEN PRETTY OFTEN 
I become excited VERY OFTEN PRETTY OFTEN 
I stutter or stammer when talking VERY OFTEN PRETTY OFTEN 
I am afraid of my father VERY OFTEN PRETTY OFTEN 


Test D-2 Questions 


I find it easy to speak out in class 

recitations VERY OFTEN 
I am unafraid of my father VERY OFTEN 
I feel well and strong VERY OFTEN 
I am unafraid of fire VERY OFTEN 
I am happy VERY OFTEN 
1 remain calm and unexcited VERY OFTEN 
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Other children are willing to play with 

me VERY OFTEN = PRETTY OFTEN 
I find school an easy place to get along in VERY OFTEN PRETTY OFTEN 
I am treated as well as other members 

of my family are treated VERY OFTEN PRETTY OFTEN 


I like school VERY OFTEN PRETTY OFTEN 
I am free from crying VERY OFTEN PRETTY OFTEN 
I keep from biting my nails VERY OFTEN PRETTY OFTEN 
People praise me VERY OFTEN PRETTY OFTEN 
I am free from anger VERY OFTEN PRETTY OFTEN 
I am unafraid of the dark VERY OFTEN PRETTY OFTEN 
I am free from feelings of wanting to run 

away from home VERY OFTEN PRETTY OFTEN 
I am free from having my feelings easily 

hurt VERY OFTEN PRETTY OFTEN 
I am unafraid during a thunderstorm VERY OFTEN PRETTY OFTEN 


I find home an easy place to get along in VERY OFTEN PRETTY OFTEN 
I feel very rested when I wake up in the 

morning VERY OFTEN PRETTY OFTEN 
I am free from day-dreaming VERY OFTEN PRETTY OFTEN 
I am free from pains in some part of 

my body VERY OFTEN PRETTY OFTEN 
I find it easy to become a leader in 

games with other children VERY OFTEN PRETTY OFTEN 
I am unafraid of high places VERY OFTEN PRETTY OFTEN 
I am free from stuttering or stammering 

when talking VERY OFTEN PRETTY OFTEN 
My parents praise me VERY OFTEN PRETTY OFTEN 
While sleeping I am free from horrible 

dreams VERY OFTEN PRETTY OFTEN 


I am free from the habit of twitching 

my neck or face VERY OFTEN PRETTY OFTEN 
I find it easy to make friends with other 

children VERY OFTEN PRETTY OFTEN 
I am free from having headaches VERY OFTEN PRETTY OFTEN 
I feel full of energy during the day VERY OFTEN PRETTY OFTEN 
I feel that my teachers treat me well 

and fairly VERY OFTEN PRETTY OFTEN 
I find it easy to do my school work right VERY OFTEN PRETTY OFTEN 
I have no fear of being followed when I 

am walking on the street VERY OFTEN PRETTY OFTEN 
I feel that people really love me VERY OFTEN PRETTY OFTEN 
I would rather play with other children 

than play by myself VERY OFTEN PRETTY OFTEN 


Test I-t Questions 


Children who feel tired when they wake in the morning are 


VERY QUEER PRETTY QUEER A LITTLE QUEER 
Children who are afraid of fire are 

VERY QUEER PRETTY QUEER A LITTLE QUEER 
Children who find it hard to make friends with other children are 

VERY QUEER PRETTY QUEER A LITTLE QUEER 
Children who often feel ill or weak are 


VERY QUEER PRETTY QUEER A LITTLE QUEER 


NEVER 
NEVER 


NEVER 


NEVER 
NEVER 
NEVER 
NEVER 
NEVER 
NEVER 
NEVER 


NEVER 
NEVER 


NEVER 


NEVER 
NEVER 


NEVER 


NEVER 
NEVER 


NEVER 
NEVER 


NEVER 


NEVER 
NEVER 
NEVER 


NEVER 


NEVER 
NEVER 


NEVER 
NEVER 


NEVER 


NOT AT ALL QUEER 


NOT AT ALL QUEER 


NOT AT ALL QUEER 


NOT AT ALL QUEER 


Childre 
are U 


Childre 
Childre 
Childré 


Childre 


Childre 
Childr 
Childr 
Childr 
Childr 
Childr 
Childr 
Childs 


Childs 


Child: 
Child: 
Child: 
Child 
Child 
Child 
Child 
Child 
Chile 
Chile 
Chile 
Chile 


Chile 
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Children who feel that other members of their family are treated better than they 


are treated are 
VERY QUEER 
Children who dislike school are 
VERY QUEER PRETTY QUEER 
Children whom other children refuse to play with are 
VERY QUEER PRETTY QUEER A LITTLE QUEER 
Children who find it hard to speak out in class recitations are 
VERY QUEER PRETTY QUEER A LITTLE QUEER 


PRETTY QUEER A LITTLE QUEER 


A LITTLE QUEER 


Children who would rather play by themselves than play with other children are 


VERY QUEER PRETTY QUEER A LITTLE QUEER 


Children who often have headaches are 

VERY QUEER PRETTY QUEER 
Children who often become gloomy or sad are 

VERY QUEER PRETTY QUEER 
Children who are afraid of their father are 

VERY QUEER PRETTY QUEER A LITTLE QUEER 
Children who feel that people often find fault with them are 

VERY QUEER PRETTY QUEER A LITTLE QUEER 
Children who are bothered with pains in some part of their body are 

VERY QUEER PRETTY QUEER A LITTLE QUEER 
Children who feel that nobody really loves them are 

VERY QUEER PRETTYQUEER A LITTLE QUEER 
Children who are afraid of being followed when walking on the street are 

VERY QUEER PRETTYQUEER A LITTLE QUEER 


A LITTLE QUEER 


A LITTLE QUEER 


Children who often feel like running away from home are 


VERY QUEER PRETTY QUEER A LITTLE QUEER 


NOT AT ALL QUEER 
NOT AT ALL QUEER 
NOT AT ALL QUEER 
NOT AT ALL QUEER 


NOT AT ALL QUEER 


NOT AT ALL QUEER 
NOT AT ALL QUEER 
NOT AT ALL QUEER 
NOT AT ALL QUEER 
NOT AT ALL QUEER 
NOT AT ALL QUEER 
NOT AT ALL QUEER 


NOT AT ALL QUEER 


Children who find it hard to become a leader in games with other children are 


VERY QUEER PRETTY QUEER A LITTLE QUEER 


Children who often cry are 
VERY QUEER PRETTY QUEER 
Children who are afraid of high places are 
VERY QUEER PRETTY QUEER A LITTLE QUEER 
Children who find it hard to do their school work right are 
VERY QUEER PRETTY QUEER A LITTLE QUEER 
Children who find school a hard place to get along in are 
VERY QUEER PRETTY QUEER 
Children who often get angry are 
VERY QUEER 


A LITTLE QUEER 


A LITTLE QUEER 


PRETTY QUEER A LITTLE QUEER 


NOT AT ALL QUEER 


NOT AT ALL QUEER 


NOT AT ALL QUEER 


NOT AT ALL QUEER 


NOT AT ALL QUEER 


NOT AT ALL QUEER 


Children who feel that their parents blame them or find much fault with them are 


VERY QUEER 
Children who often become excited are 
VERY QUEER PRETTYQUEER A LITTLE QUEER 
Children who often have horrible dreams while sleeping are 
VERY QUEER PRETTY QUEER A LITTLE QUEER 
Children who find home a hard place to get along in are 
VERY QUEER PRETTY QUEER 
Children who often day-dream are 
VERY QUEER 
Children who are afraid of the dark are 
VERY QUEER PRETTY QUEER 
Children who are afraid during a thunder-storm are 
VERY QUEER PRETTY QUEER 
Children who bite their nails are 
VERY QUEER 


PRETTY QUEER A LITTLE QUEER 


A LITTLE QUEER 
PRETTY QUEER A LITTLE QUEER 
A LITTLE QUEER 
A LITTLE QUEER 


PRETTY QUEER A LITTLE QUEER 


NOT AT ALL QUEER 
NOT AT ALL QUEER 
NOT AT ALL QUEER 
NOT AT ALL QUEER 
NOT AT ALL QUEER 
NOT AT ALL QUEER 
NOT AT ALL QUEER 


NOT AT ALL QUEER 
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Children who often feel tired during the daytime are 
VERY QUEER 
Children whose feelings are easily hurt are 
VERY QUEER 
Children who feel that their teachers treat them badly or unfairly are 
VERY QUEER 
Children who stutter or stammer when talking are 
VERY QUEER 
Children who have the habit of twitching their neck or face are 
VERY QUEER 


VERY MANY 


VERY MANY 


VERY MANY 


VERY MANY 


VERY MANY 


VERY MANY 


VERY MANY 


VERY MANY 


VERY MANY 


VERY MANY 


VERY MANY 


VERY MANY 


VERY MANY 


VERY MANY 


VERY MANY 


VERY MANY 


VERY MANY 


VERY MANY 


VERY MANY 


VERY MANY 
VERY MANY 


VERY MANY 
VERY MANY 
VERY MANY 


VERY MANY 


VERY MANY 


PRETTY MANY 


PRETTY MANY 


PRETTY MANY 


PRETTY MANY 


PRETTY MANY 


PRETTY MANY 


PRETTY MANY 


PRETTY MANY 


PRETTY MANY 


PRETTY MANY 


PRETTY MANY 


PRETTY MANY 


PRETTY MANY 


PRETTY MANY 


PRETTY MANY 


PRETTY MANY 


PRETTY MANY 


PRETTY MANY 


PRETTY MANY 


PRETTY MANY 
PRETTY MANY 


PRETTY MANY 
PRETTY MANY 
PRETTY MANY 


PRETTY MANY 


PRETTY MANY 


FEW 


FEW 
FEW 
FEW 
FEW 


FEW 
FEW 
FEW 
FEW 


PRETTY QUEER 


PRETTY QUEER 


PRETTY QUEER 


PRETTY QUEER 


PRETTY QUEER 


NO 


NO 
NO 
NO 
NO 


NO 
NO 


NO 


ALITTLE QUEER NOTAT ALL QUE 


A LITTLE QUEER NOT AT ALL QUER 
A LITTLE QUEER NOT AT ALL QUEER 
A LITTLE QUEER NOT AT ALL QUER 


A LITTLE QUEER NOT AT ALL QUEER 


Test I-2 Questions 


children feel tired when they wake up in the 
morning. 

children bite their nails. 

children often get angry. 

children are afraid of fire. 

children find it hard to speak out in clas 
recitations. 

children often feel tired during the daytime. 

children often become gloomy or sad. 

children are afraid of high places. 

children are afraid of being followed when 
they are walking on the street. 


children feel that their teachers treat them 
badly or unfairly. 

children are bothered with pains in some part 
of their body. 

children would rather play by themselves 
than play with other children. 

children find home a hard place to get along 
in. 

children often day-dream. 

children are afraid of their fathers. 

children find it hard to make friends with 
other children. 

children have other children refuse to play 
with them. 

children find it hard to do their school work 
right. 


children have the habit of twitching thei 
neck or face. 

children feel that nobody really loves them. 

children often feel like running away from 
home. 

children often feel ill or weak. 

children often cry. 

children often have horrible dreams while 
sleeping. 

children find school a hard place to get along 
in. 

children feel that other members of thei! 
family are treated better than they art 
treated. 


VERY MA\ 


VERY M A 





class 


time, 


when 


them 
* part 
selves 


along 


while 
long 


thei! 
are 


yERY MANY 


yERY MANY 
VERY MANY 


VERY MANY 
VERY MANY 
yERY MANY 
vERY MANY 
VERY MANY 
VERY MANY 
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PRETTY MANY 


FEW 


NO 


children find it hard to become a leader in 
games with other children. 


children feel that their parents blame them 
or find much fault with them. 

children stutter or stammer when talking. 

children fee] that people find much fault 
with them. 

children often become excited. 

children are afraid of the dark. 

children dislike school. 

children are afraid during a thunder-storm. 

children often have headaches. 

children’s feelings are easily hurt. 





